An image paragraph description text generation method based on information entropy

Abstract

The application discloses an image paragraph description text generation method based on information entropy, constructs an image paragraph description text generation model, and the model comprises a feature extraction network, a feature fusion network and a text decoding network; for a graph-text data set, image visual information features, image semantic features, image spatial position features and text content features are extracted and interactively fused, and when decoding text at each time step, the features are dynamically focused and fused according to generated content; the model is trained by using cross-entropy loss, and similarity penalty is applied in the decoding generation process to inhibit repeated phrases and ensure alignment with the text distribution of the data set; information entropy of the paragraph text is introduced for quantitative measurement, the information entropy index is used as an additional reward signal, and the model is trained by using reinforcement learning based on a policy gradient; and the model after completing the reinforcement learning training is used to generate image paragraph description text. The application can improve the ability of describing image details and enhance the diversity of paragraph description content under the premise of ensuring accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, and relates to a method for generating image paragraph descriptive text, particularly a method for generating image paragraph descriptive text based on information entropy. Background Technology

[0002] Image paragraph description text generation is an interdisciplinary task in computer vision image understanding and natural language generation processing. The goal is to use algorithms to understand images and generate logically coherent text for each image, ensuring the content is detailed, diverse, consistent, and coherent. This method enables cross-modal conversion of image-to-text information. In e-commerce, it can automatically generate detailed descriptions for merchants' products, reducing the cost and workload of manual description. It can also be applied to areas such as enhanced image semantic retrieval.

[0003] Cross-modal generation from images to long text typically employs an encoder-decoder framework. First, a series of regions of interest features in the image are extracted using a convolutional neural network and a region proposal network in the encoder. In the text generation stage, the global semantic feature vector of the image is used as the input to the decoder. At each time step, the state vector of the decoder's hidden layer is mapped to the entire dictionary through a fully connected layer. The word with the highest mapping probability is used as the generation result, and a complete sentence is generated in a temporal manner.

[0004] In existing technologies, the paper [Krause J, Johnson J, Krishna R, et al. A Hierarchical Approach for Generating Descriptive Image Paragraphs [C] / / 2017 IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2017:3337-3345.] adopts an Encoder-Decoder structure and proposes a hierarchical image paragraph description model, Hierarchical-RNN, based on the hierarchy of paragraphs, sentences, objects, actions, and words in the dataset. It first generates a topic vector for each sentence and then generates each word from it. The paper [Chatterjee M, Schwing A G. Diverse and coherent paragraph generation from images [C] / / Proceedings of the European Conferenceon Computer Vision (ECCV). 2018:729-744.] introduces a coherence vector to improve the logic and coherence between sentences, guiding paragraph generation. Its model further generates a global topic vector in the sentence RNN, and adds the hidden layer output of the previous word RNN and the global vector to the input of the word RNN, coupling the two to obtain a coherent vector, thereby ensuring a smooth transition between sentences. [Liang X, Hu Z, Zhang H, et al. Recurrent topic-transition gan for visual paragraph generation[C] / / Proceedings of the IEEE international conference on computer vision.2017:3362-3371.] proposes a semi-supervised paragraph generation framework, RTT-GAN, a generative adversarial network. The quality of sentences generated by the paragraph generator is evaluated by a multi-level paragraph discrimination network, which includes sentence-level rationality and topic transition coherence. In addition, compared with the image features lost in the Hierarchical-RNN region pooling process, RTT-GAN focuses on local regions through an attention mechanism, which also improves the semantic alignment between images and paragraphs.[Yang X, Gao C, Zhang H, et al. Hierarchical scene graph encoder-decoder for image paragraph captioning[C] / / Proceedings of the 28th ACM International Conference on Multimedia.2020:4181-4189.] The attention mechanism used no longer focuses solely on object features, but rather on the features of the extracted scene graph and sub-scene graphs, achieving fine-grained image description and increasing the CIDEr value, an metric for evaluating paragraph captioning models, to 36.02.

[0005] Despite significant technological advancements in image paragraph description generation, existing methods still suffer from several shortcomings in terms of the quality of generated text descriptions. Specifically, these methods often produce paragraphs with unnecessary repetition of words and phrases, excessive information redundancy, and a lack of diversity in content presentation, failing to adequately capture and convey the key details of the image. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention aims to provide an image paragraph description text generation method based on information entropy, so as to improve the ability to describe image details and enhance the diversity of paragraph description content while ensuring accuracy.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for generating image paragraph description text based on information entropy includes the following steps:

[0009] Step 1: Construct an image paragraph description text generation model, which includes a feature extraction network, a feature fusion network, and a text decoding network; for the image-text dataset, use the feature extraction network to extract image visual information features, image semantic features, image spatial location features, and text content features;

[0010] Step 2: Using the feature fusion network, the features extracted in Step 1 are interactively fused.

[0011] Step 3: Using the text decoding network, dynamically focus on the fusion features based on the generated content when decoding the text at each time step;

[0012] Step 4: Train the image paragraph description text generation model using cross-entropy loss, and apply similarity penalty during the decoding and generation process to suppress repeated phrases and ensure alignment with the text distribution of the dataset;

[0013] Step 5: Introduce a quantitative measure of information entropy of paragraph text, and use the information entropy index as an additional reward signal to train the model using reinforcement learning based on policy gradient.

[0014] Step 6: Using the image paragraph description text generation model that has completed reinforcement learning training, generate image paragraph description text with the image as input.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] Compared with existing image paragraph description techniques, this invention focuses on more comprehensive image details during the image feature extraction stage and explicitly extracts target semantic and spatial location information from the image, which helps improve the accuracy of the description. In the construction of the reward function, a quantitative metric for the diversity of paragraph text information is proposed. Guided by information entropy rewards, the text generated by the image paragraph description model can cover more details of the image while ensuring accuracy, thus improving the diversity of paragraph text. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the method of the present invention.

[0018] Figure 2 This is a block diagram of the image paragraph description text generation model based on information entropy in this invention.

[0019] Figure 3 Example diagrams for generating image text descriptions in this invention.

[0020] Figure 4 To and Figure 3 Image and text descriptions are translated one-to-one into Chinese. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0022] refer to Figure 1 and Figure 2This invention discloses an image paragraph description text generation model method based on information entropy, which mainly includes the following five stages: image and text feature extraction, feature fusion, text decoding and generation, maximum likelihood optimization, and information entropy optimization. This method extracts visual information features, semantic features, and spatial location features from the image, and extracts text content features. It uses a Transformer structure that effectively accommodates both image and text modal information to fuse different modalities. In the first stage of text decoding, cross-entropy loss is used to train the model. In the second stage, during reinforcement learning training based on policy gradients, a quantitative measure of the information entropy of the paragraph text is introduced, using this indicator as an additional reward signal to encourage the model to generate richer and less repetitive text content. Experiments conducted on the image paragraph description dataset provided by Stanford University demonstrate that this method shows significant effects in improving the accuracy and text diversity of paragraph descriptions.

[0023] The present invention specifically includes the following steps:

[0024] Step 1: Extract visual information features from the image.

[0025] First, the image resolution is scaled and cropped to a uniform size. A convolutional neural network is then used to obtain P image patches. In this embodiment, the image resolution scaling and cropping size is chosen to be 480×480, and the image patch size is 16×16. A deep residual network is then used to extract image features. Specifically, the first three blocks of the network are used; these are relatively shallow layers in a deep residual network. Compared to the entire network, they learn low-level features and some local features of the input image, typically including basic information such as edges and textures. In this embodiment, ResNet152 is chosen as the deep residual network. The advantage of the first three blocks is that they can capture subtle details of the input image and extract relatively general feature representations. Then, a visual codebook trained by clustering the training dataset is used. Performing image vector quantization operation q yields the final image representation, mapping high-dimensional image data to a low-dimensional discrete codebook. This simplifies image representation and reduces computational complexity. The nearest neighbor reconstruction formula is as follows:

[0026]

[0027] In the formula, I i The image visual information features obtained by mapping, z is the initial feature of the i-th image obtained by the deep residual network. k Let n be the k-th codebook vector, where K is the number of codebook vectors. z is the dimension of the codebook vector, and w and h represent the width and height of the entire image.

[0028] Step 2: Extract semantic features and spatial location features of the image.

[0029] Using the Faster R-CNN model framework, the bounding boxes containing the target proposed by the Region Proposal Network (RPN) are processed. Fixed-size feature maps after region pooling (ROI pooling) and a fully connected object classifier are then used to obtain the semantic labels y = (y1,...,y2) of the target in the image. N Then, the image semantic features O = (O1,...,O) are obtained through a Transformer-based bidirectional encoder-decoder architecture language model. N For example, salient objects in the image such as "train", "men", and "cloud" are used to obtain object vector features using a BART language model based on a Transformer bidirectional encoder-decoder architecture. The corresponding location information of the target is obtained through bounding box regression and encoded as image spatial location features. The calculation method is as follows:

[0030]

[0031] In the formula, x i y i The x and y coordinates of the center point of the i-th target bounding box are represented by w. i h i Let represent the width and height of the i-th target bounding box; w and h represent the width and height of the entire image; N is the number of target semantic labels; y N Represents the Nth target semantic label, O N This represents the Nth image semantic feature.

[0032] Step 3: Extract text content features.

[0033] For the image-text pairs in the dataset, since they were manually labeled by volunteers, the text description dataset was first cleaned. In this invention, Byte Pair Encoding (BPE) and other methods are first used to segment the text sequences in the image-text dataset into fine-grained sub-word or sub-word fragment sequences to solve the vocabulary size limitation problem and improve the ability to handle unseen words. The main idea of ​​BPE is to construct a vocabulary by iteratively merging the most frequently occurring characters or character combinations, which can improve the model's ability to understand complex text. Based on this, the text embedding vector representation is obtained by embedding sub-words using a pre-trained embedding model. Then, positional encoding and fragment encoding are added to the embedding vector. Positional encoding is used to provide the model with information about the position of words in the sequence. Here, trigonometric function absolute positional encoding is used. Compared with relative positional encoding, absolute positional encoding provides more accurate positional information, allowing the model to consider both semantic and positional information of words when processing the input sequence. In this invention, fragment encoding distinguishes different segments or sentences. This is achieved by assigning an identifier (usually a special label such as [CLS], [SEP]) to each segment in the text's embedding vector representation, and then adding or concatenating this identifier with the embedding of the corresponding segment in the input sequence (i.e., the text's embedding vector representation). In this way, the model can distinguish different segments through fragment encoding, thereby learning their features and relationships. Finally, the embedding vector representation, along with the corresponding positional and fragment encodings, are added and processed through a pre-trained language model, BART, to obtain fixed-dimensional text content features T. i .

[0034] Step 4: Effectively fuse image visual information features, image semantic features, and image spatial geometric information features.

[0035] By representing the image and text features through steps 1, 2, and 3, this invention performs a nonlinear transformation on the features extracted in each step before concatenating them. Specifically, the image visual information features, image semantic features, and image spatial location features employ a first absolute position encoding, while the text content features employ a second absolute position encoding to preserve temporal information, thus obtaining the input feature x. The calculation formula is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] x = [u I ,u O ,u P ,u T ]

[0041]

[0042]

[0043] In the formula, X represents the input vector, W1 and W2 represent the weight vectors of the linear matrix, and b1 and b2 represent the bias vectors; I i O represents the visual information feature of the i-th image. i Let P represent the semantic feature of the i-th image. i T represents the spatial location feature of the i-th image. i This represents the i-th text content feature. I u O u P u T These are, respectively, image visual information features, image semantic features, image spatial location features, and text content features mapped to a defined dimensional embedding space. This represents four structurally identical, parameter-independent multilayer perceptron networks. Each perceptron network consists of a fully connected layer and a regularized nonlinear activation function GELU, enhancing the model's nonlinearity and generalization capabilities. The input feature x obtained after feature concatenation is fed into the feature fusion part, which uses a Transformer encoder as its backbone network. Specifically, this invention uses PreLayerNormalization, placing the LayerNormalization layer before the residual connections and feedforward neural network layers in the encoder, making the encoder easier to train and converge. This is represented as:

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] In the formula, MultiHeadAtt is a multi-head attention layer with 12 attention heads, a hidden layer dimension of 1024, a feedforward neural network intermediate layer dimension of 4096, and a Transformer stacking layer of 12. This is the input at the i-th position of the l-th Transformer layer. W represents the output after passing through the LayerNormalization layer, multi-head attention layer, residual connection layer, LayerNormalization layer, and activation function layer, respectively. 1,l and W 2,l b represents the weights of a linear matrix. 1,l and b 2,l Indicates bias.

[0051] Step 5, text decoding.

[0052] This step uses a standard Transformer decoder network for text decoding. The decoding process is as follows:

[0053]

[0054]

[0055] In the formula, query is the text prompt question input to the model, such as "what does the image describe?", x encoder The output is the result after passing through the model encoder, where p represents the probability distribution of the hidden state on the vocabulary obtained through nonlinear transformation and softmax operation. The linear matrix weights W1 and W2, and the biases b1 and b2 are all learnable parameters. Sub-words are selected as the generation result according to the decoding strategy, and complete sentences are generated sequentially. The Decoder has 12 stacked Transformer layers, and the total number of parameters in the paragraph description generation model is 470M. Regarding the decoding strategy, the baseline method uses greedy search, selecting the word or character with the highest probability output by the current model as the next prediction. Two schemes are used for prediction: Scheme 1: Random sampling, which randomly selects the next word or character based on the probability distribution of the model output. Specifically, the selection probability of each word or character corresponds to the probability of the model output. Random sampling can produce more diverse outputs, but the disadvantage is that the generated text may not be accurate or coherent enough. Scheme 2: Nucleus Sampling (also known as Top-pSampling), an improvement on random sampling, which also limits the selection range of the next prediction. Specifically, it sorts the probability distributions of the model outputs cumulatively and selects the smallest set of words or characters whose cumulative probabilities exceed a pre-set threshold (e.g., 0.9). Then, it randomly samples from this limited set of words or characters to select the next prediction, balancing diversity and accuracy during decoding.

[0056] It is worth noting that, to achieve the feature extraction in steps 1 to 3, the feature fusion in step 4, and the text decoding in step 5, the image paragraph description text generation model constructed in this invention includes a feature extraction network, a feature fusion network, and a text decoding network. For the image-text dataset, the feature extraction network extracts image visual information features, image semantic features, image spatial location features, and text content features. The feature fusion network then interweaves the extracted features, and the text decoding network dynamically focuses on the fused features based on the generated content at each time step when decoding the text.

[0057] Step 6: Train the above image paragraph description text generation model using cross-entropy loss, and apply similarity penalty during the decoding and generation process to suppress repeated phrases and ensure alignment with the text distribution of the dataset.

[0058] Cross-entropy loss is used for model training. The calculation method for cross-entropy loss is as follows:

[0059]

[0060] In the formula, N represents the batch size during training, M represents the number of text segments, and l ij p represents the tag corresponding to the j-th word of the i-th sample. ij This represents the probability of the j-th word segmentation for the i-th sample predicted by the model. Its purpose is to ensure the alignment of the model output with the dataset distribution, guiding the model to learn and generate accurate descriptions. Since model predictions are prone to repetition when generating long texts, especially paragraphs, a similarity penalty is introduced. Specifically, at time step t, consecutive triples from the previous t-1 time steps are retained, and the word probability penalty is calculated by determining the maximum similarity between the current triple and the preceding triples. The calculation method is as follows:

[0061]

[0062] In the formula, V represents the vocabulary, x ＜t This represents the word segmentation sequence generated before the t-th time step, where s is the similarity calculation method, and h represents the triplet. v This represents the triple formed by the v-th word and time steps t-1 and t-2. This indicates that the j-th word x has been generated. j The triplet formed by time steps j-1 and j-2, where α is a hyperparameter representing the degree of suppression of duplicate triplets. When α is 0, it means that no similarity penalty is applied, and when α approaches 1, it means that duplicate triplets are completely suppressed.

[0063] Step 7 introduces a quantitative measure of information entropy for paragraph text, using the information entropy index as an additional reward signal to train the model through policy gradient-based reinforcement learning. The image paragraph description text generation model, having completed reinforcement learning training, generates image paragraph description text from images as input.

[0064] The training process of image description models suffers from a mismatch between training and evaluation objectives: the quality of an image description model is ultimately evaluated using metrics such as BLEU and CIDEr. However, during training, after the decoder predicts the model distribution, maximum likelihood is typically used to estimate and optimize the model, leading to a mismatch between the training and evaluation objectives. Furthermore, because these evaluation metrics are non-differentiable, gradient-based methods cannot be used for backpropagation and model optimization. This invention defines an information entropy optimization model to improve the accuracy and contextual diversity of generated paragraph text, thereby increasing the coverage of image content description. Specifically, using a policy gradient-based reinforcement learning method, the image paragraph description text generation model trained through the aforementioned steps is treated as an agent. A reward function is designed for the generated paragraph description text, and the training objective is transformed into maximizing the expectation, i.e.

[0065]

[0066] In the formula w s The generated paragraph description is represented by r, which represents the paragraph's reward function, and p is the value of p. θ To obtain the generated paragraph text sequence w for the model s The probability of . Using the Monte Carlo sampling method, the gradient of the training target is simplified to .

[0067]

[0068] To ensure stable and rapid convergence during training, a baseline method is introduced to obtain the reward r. b The gradient of the training objective is as follows:

[0069]

[0070]

[0071] If the currently sampled word is better than the word generated during the testing phase, then the value of the above formula is negative in the dimension of this word, the gradient will increase, thus improving the score of this word; while for other words, the latter term is positive, the gradient will decrease, thus decreasing the scores of other words. In this training process, the traditional approach is to use the CIDEr value, which evaluates the similarity between the generated paragraph and the n-tuples of paragraph labels in the dataset, as the reward function r. This invention introduces paragraph information entropy as a supplement to the reward function. Information entropy is used to characterize the amount of information in the paragraph content to improve the diversity of the model's output text. Information entropy can reflect the distribution of the probability of the words it contains. Specifically, in information entropy, H(X) = -∑ x∈X The information entropy of a paragraph is p(x)logp(x), with a maximum value of NlogN, and increases with the increase of N, i.e., the increase of the number of different words in the paragraph. Assuming p is the probability of a word in the paragraph, this means that if a paragraph contains a large mixture of common and rare words, its information entropy will be higher than that of a paragraph containing only common words. Therefore, this invention uses it as a representation of the diversity of paragraph content. The formula for calculating paragraph information entropy is as follows:

[0072]

[0073] In the formula, p(y i ) represents the i-th word y in the paragraph. i The probability of a word is denoted by , where n represents the number of unique words, and c is the paragraph length, used for regularization to prevent the model from generating excessively long texts. Otherwise, repeatedly generating the same sentence would not result in unchanged information entropy, leading to repetitive descriptions in long text outputs. Calculating the information entropy of a paragraph allows us to assess the complexity of the word probability distribution within it. A higher information entropy in a paragraph indicates that it contains more diverse and balanced words, resulting in richer content diversity. The final reward function and baseline reward r are then discussed. b The designs are respectively

[0074] r(Y) = CIDEr(Y) + βη(Y)

[0075]

[0076] In the formula, β is the reward coefficient, and CIDEr(Y) is the industry-standard evaluation metric calculated by comparing the generated paragraph Y with the dataset labels. Each forward propagation samples F paragraph descriptions, b m The baseline reward for the m-th generated paragraph is the average reward of the remaining F-1 paragraphs excluding the m-th paragraph. Compared to using the paragraph reward as the baseline in greedy search, this method has smaller gradient variance, stronger convergence, and better robustness. Figure 3 This is an example image illustrating the generation of image paragraph description text in this invention. Figure 4 yes Figure 3 The corresponding Chinese translation reveals that while the OFA-CIDEr model generates more details than the ground truth, it also generates repetitive phrases, such as "metal" and "silver" in image 2, "surfboard," "wave," and "water" in image 3, and "container" in image 4. This indicates that the model tends to forget already generated content when generating paragraph text, and its exploration of the diversity of objects within the content is insufficient. After optimization with the Entropy reward, the model can describe more details, such as objects like "helmets," "cloud," "stripes," "foam," and "carpet," and attributes like "orange" and "choppy." Furthermore, the attention to detail brought by the entropy reward can help improve the accuracy of content description. For example, in image 2, the model detects "person," when it should actually be "tree" and "stop sign" generated by OFA-Entropy; in image 3, "man" is incorrectly described as "woman."

[0077] As can be seen, the text generated by the model of this invention can cover more details of the image while ensuring accuracy, thus improving the diversity of paragraph text.

Claims

1. A method for generating image paragraph description text based on information entropy, characterized in that, Includes the following steps: Step 1: Construct an image paragraph description text generation model, which includes a feature extraction network, a feature fusion network, and a text decoding network; for the image-text dataset, use the feature extraction network to extract image visual information features, image semantic features, image spatial location features, and text content features; Step 2: Using the feature fusion network, the features extracted in Step 1 are interactively fused. Step 3: Using the text decoding network, dynamically focus on the fusion features based on the generated content when decoding the text at each time step; Step 4: Train the image paragraph description text generation model using cross-entropy loss, and apply similarity penalty during the decoding and generation process to suppress repeated phrases and ensure alignment with the text distribution of the dataset; the cross-entropy loss is calculated as follows: In the formula, N Indicates the batch size during training. M Indicates the number of text segments. Indicates the first i The first sample j Each word segment corresponds to a tag. The model predicts the first i The first sample j Each word segmentation probability; The similarity penalty is as follows: at time step t, consecutive triples from the previous t-1 time steps are retained, and the word probability penalty is calculated by determining the maximum similarity between the current triple and the previous triples. The calculation method is as follows: In the formula, V Indicates a vocabulary list, Indicates the first t The word segmentation sequence generated before each time step s This is the similarity calculation method. h Represents a triple. Indicates the first v Each word and t -1、 t Triples consisting of -2 time steps Indicates the number of generated [items]. j Word and j -1、 j Triples consisting of -2 time steps This is a hyperparameter representing the degree to which duplicate triples are suppressed. A value of 0 indicates that no similarity penalty is applied. A value approaching 1 indicates complete suppression of repeated triples; Step 5: Introduce a quantitative measure of information entropy for the paragraph text. Use the information entropy index as an additional reward signal to train the model using policy gradient-based reinforcement learning. The method is as follows: The image paragraph description text generation model obtained after training in step 4 is regarded as an agent. A reward function is designed for the generated paragraph description text, and the training objective is transformed into maximizing the expectation, i.e. In the formula This indicates the generated paragraph description. r The reward function for the paragraph. To generate paragraph text sequences for the model The probability of the reward obtained by introducing the baseline method. The gradient of the training objective is: Information entropy is introduced as a supplement to the reward function. This information entropy characterizes the amount of information in a paragraph, using the probability of words within the paragraph as a representation of the paragraph's content diversity. The formula for calculating paragraph information entropy is as follows: In the formula, Indicates the first paragraph i one word The probability, n Indicates the number of unique words. c The length of the paragraph, the final reward function, and the baseline reward are respectively: In the formula, As the reward coefficient, To generate paragraphs Y The industry-standard evaluation metrics are calculated by comparing the dataset labels with each forward propagation sampling. F Paragraph description, For the first m The baseline reward for the generated paragraph is obtained by removing the first paragraph. m The remaining paragraphs F The average reward of -1 paragraphs is used as the baseline reward; Step 6: Using the image paragraph description text generation model that has completed reinforcement learning training, generate image paragraph description text with the image as input.

2. The image paragraph description text generation method based on information entropy according to claim 1, characterized in that, In step 1, the method for extracting visual information features of the image using the feature extraction network is as follows: The image resolution is scaled and cropped to a uniform size, and then a convolutional neural network is used to obtain... P For each image patch, the first three blocks of a deep residual network are used to extract preliminary image features. Then, the visual codebook trained by clustering the training dataset was used. Perform image vector quantization operation By mapping high-dimensional image data to a low-dimensional discrete codebook, the visual information features of the image are obtained. The calculation formula is: In the formula, The first one obtained for deep residual networks i Preliminary features of an image For the first k codebook vectors K The number of codebook vectors. The dimension of the codebook vector. , This indicates the width and height of the entire image.

3. The image paragraph description text generation method based on information entropy according to claim 1, characterized in that, In step 1, the method for extracting semantic features and spatial location features of the image using the feature extraction network is as follows: Using the Faster R-CNN model framework, the bounding boxes containing the target proposed by the Region Proposal Network (RPN) are processed, and the fixed-size feature maps after region pooling and a fully connected layer object classifier are used to obtain the semantic labels of the targets in the image. Then, the semantic features of the image are obtained through a language model based on a Transformer-based bidirectional encoder-decoder architecture. The target's location information is obtained using bounding box regression and encoded as image spatial location features. The calculation method is as follows: In the formula, , Indicates the first The x and y coordinates of the center point of the target bounding box; , Indicates the first Width and height of the target bounding box; , This represents the width and height of the entire image. U The target number of semantic tags, Indicates the first U One target semantic label, Indicates the first U Image semantic features.

4. The image paragraph description text generation method based on information entropy according to claim 1, characterized in that, In step 1, the method for extracting text content features using the feature extraction network is as follows: Segment the text sequence in the image-text dataset into fine-grained sub-word or sub-word fragment sequences; The embedding vector representation of the text is obtained by embedding subwords using a pre-trained embedding model; Add positional encoding and fragment encoding to the embedded vector representation; The embedded vector representation, along with the corresponding positional and fragment encodings, are summed and then passed through a pre-trained language model, BART, to obtain fixed-dimensional text content features. .

5. The image paragraph description text generation method based on information entropy according to claim 4, characterized in that, The position encoding is a trigonometric function absolute position encoding; the segment encoding is achieved by assigning an identifier to each segment in the embedded vector representation of the text, and then adding or concatenating the identifier with the embedded vector representation of the text.

6. The image paragraph description text generation method based on information entropy according to claim 1, characterized in that, Step 2: Using the feature fusion network, the features extracted in Step 1 are interactively fused, as follows: The features extracted in step 1 are nonlinearly transformed and then concatenated. Specifically, the image visual information features, image semantic features, and image spatial location features are encoded using the first absolute position encoding, while the text content features are encoded using the second absolute position encoding, in order to preserve temporal information, thus obtaining the input features. x The calculation formula is as follows: In the formula, , , , These are, respectively, image visual information features, image semantic features, image spatial location features, and text content features mapped to a defined dimensional embedding space; , , , This represents four multilayer perceptron networks with identical structures and independent parameters. Indicates the first i Image visual information features, Indicates the first i Image semantic features, Indicates the first i Image spatial location features, Indicates the first i Each text content feature; the input feature obtained by concatenating the features. x The input is fed into a feature fusion network with a Transformer encoder as the backbone network.

7. The image paragraph description text generation method based on information entropy according to claim 6, characterized in that, The multilayer perceptron network consists of a fully connected layer and a regularized nonlinear activation function GELU in each layer. In the formula, X Represents the input vector. , Represents the weights of a linear matrix. , Indicates bias; The feature fusion network uses a Transformer encoder as its backbone and employs PreLayerNormalization, which places the LayerNormalization layer before the residual connections and feedforward neural network layers in the encoder. This makes the encoder easier to train and converge, as shown below: In the formula, MultiHeadAtt is a multi-head attention layer with 12 attention heads, a hidden layer dimension of 1024, a feedforward neural network intermediate layer dimension of 4096, and a Transformer stacking layer of 12. For the first l Transformer layer i Input at each position, ~ These represent the outputs after passing through the LayerNormalization layer, the multi-head attention layer, the residual connection layer, and the activation function layer, respectively. and Represents the weights of a linear matrix. and Indicates bias.

8. The image paragraph description text generation method based on information entropy according to claim 1, characterized in that, Step 3: The text decoding network adopts a standard Transformer decoder network, and the decoding process is as follows: In the formula, query For the text input prompt question, This is the output after passing through the model encoder. p The probability distribution over the vocabulary is obtained by applying nonlinear transformations and a softmax operation to the hidden states, with linear matrix weights. , and bias , All parameters are learnable. Sub-words are selected as the generation result according to the decoding strategy, and complete sentences are generated in a temporal manner. The word or character with the highest output probability is selected as the next prediction.

Images