Diversified image description generation method based on image semantic segmentation guidance

Abstract

The invention discloses a diversified image description generation method based on image semantic segmentation guidance, and provides a diversified image description method based on image semantic segmentation guidance and conditional variation Transform. According to the method, image semantic segmentation features are fused to provide target semantic guidance, visual decoupling is effectively carried out on target detection features, and more accurate visual features are identified. In addition, a conditional variation Transform is introduced to learn a one-to-many mapping relation between an image and a text by optimizing an ELBO objective function, and diversified image description generation is realized. A large number of experimental results prove that the method is obviously superior to current mainstream diversified image description models in accuracy and diversity indexes.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular to a method for generating diversified image descriptions guided by image semantic segmentation. Background Art

[0002] At present, the Transformer framework has become a common model in the field of image description. In traditional image description tasks, a one-to-one mapping from image to sentence is usually learned. Since only the accuracy of sentence generation is concerned, the generated sentences are mostly repetitive and single. To address this problem, researchers have proposed a diversified image description task, which aims to generate multiple sentences for an image. Each sentence must ensure both the accuracy of the content and its own diversity. For models trained using traditional methods such as MLE and SCST, the standard Transformer generates overly common words or phrases because it only focuses on accuracy indicators, and only generates one description for the image, which makes it difficult to fully summarize the image content. At the same time, most current image description generation models use features extracted by the target detection model FasterR-CNN as visual features, lacking semantic information guidance, and the generated descriptions are not accurate and comprehensive enough. Summary of the invention

[0003] The purpose of the present invention is to provide a diversified image description generation method guided by image semantic segmentation to solve the problems raised in the above background technology.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: In order to achieve the above object, the present invention adopts the following technical solution: A diversified image description generation method based on image semantic segmentation guidance, characterized in that it includes: step S100, for a given input image I, obtaining an image feature F, an image semantic segmentation target feature M and a text description X of the input image I;

[0005] Step S200: Use the pre-trained FasterR-CNN and mask feature extractor to process the target detection feature F and the image semantic segmentation target feature M to obtain the final encoding feature F of the input image I. N 、M N , and will be combined into V N ={F N ,M N};

[0006] Step S300, based on the posterior inference sub-network and the prior approximation sub-network in the conditional variational decoding network, the encoding feature combination V N Learn with the text description X to obtain the posterior latent variables of the text Prior hidden variables Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q ;

[0007] Step S400, based on the generator according to the hidden variable s q , interactive features between text and vision F q , combination V N Get an image description.

[0008] The specific process of step S100 includes:

[0009] Step S101, obtaining an image feature F of an image I through a pixel encoder;

[0010] Step S102, obtaining pixel features of image feature F through a pixel decoder;

[0011] Step S103, defining a cross attention mechanism of a pixel decoder through a kMax decoder, and the pixel decoder obtains a mask embedding vector;

[0012] Step S104, multiplying the pixel feature and the mask embedding vector to obtain a predicted mask S;

[0013] Step S105, by performing a mask-based pooling operation on the image feature F and the predicted mask S, the image semantic segmentation target feature M is obtained, where H and W are the height and width of the image I, N is the number of segmented regions, h and w are the height and width of the image feature F, D M The feature dimension representing the features of each mask region.

[0014] In step S105, for each region of the image I, under the guidance of the prediction mask S, the image semantic segmentation target feature M is calculated by pooling the image feature F

[0015] M(N,D M )=Σ hw S(N,h,w)·F(h,w,D M ) (1)

[0016] Where S = {s1,s2,...,s N},s i ∈{0,1} H×W Indicates whether each pixel belongs to the i-th region. If s i =1, it means it belongs to the i-th region, otherwise it does not belong to it; H and W are the height and width of image I, N is the number of segmented regions, h and w are the height and width of image feature F, D M The feature dimension representing the features of each mask region.

[0017] Step S200 includes:

[0018] Step S201: input the image target feature F and the image semantic segmentation target feature M into their respective linear layers, and map the feature dimension to d model , and get the feature F 0 、M 0 ;

[0019] Step S202: feature F 0 、M 0 Input into the standard Transformer encoder to obtain the final encoding feature F N 、M N ;in

[0020] The output of the nth layer in the Transformer encoder is input into the multi-head attention module of the n+1th layer for residual normalization

[0021]

[0022]

[0023] Residual normalization followed by a feed-forward network and another residual normalization

[0024]

[0025]

[0026] Where AddNorm(·) represents the residual normalization operation.

[0027] The conditional variational decoding network in step S300 includes a posterior inference sub-network and a priori approximation sub-network; wherein

[0028] Methods in the posterior inference subnetwork include:

[0029] Step S301: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ;

[0030] Step S302: Input vector W 0 Input to the multi-head attention module for self-attention encoding and then perform residual normalization to obtain the text feature W q

[0031] W q =AddNorm(MultiHead(W 0 ,W 0 ,W 0 )) (6)

[0032] Step S303: Use a multi-head cross attention module to analyze the text feature W q and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. q

[0033] F q =AddNorm(MultiHead(W q ,V N ,V N )) (7)

[0034] Step S304: Processing conditional distribution using feedforward network Get the posterior latent variable of the word

[0035]

[0036] in, is the hidden variable of the previous word, is the time-dependent mean μ t and standard deviation σ t Multivariate Gaussian distribution of ;

[0037] The methods in the prior approximate subnetwork include:

[0038] Step S311: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ;

[0039] Step S312: Input vector W 0 Input to the masked multi-head attention module to obtain the text feature W p

[0040] W p =AddNorm(MaskedMultiHead(W 0 ,W 0 ,W 0 )) (9)

[0041] Step S313: convert the text feature W p Input to the multi-head cross attention module and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. p

[0042] F p =AddNorm(MultiHead(W p ,VN ,V N )) (10)

[0043] Step S314, using the feedforward network to process the conditional distribution Get the prior latent variables of words

[0044]

[0045] Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q .

[0046] The multi-head cross-attention module in step 313 shares the weights with the multi-head cross-attention module in step S303.

[0047] Step S400 includes:

[0048] Step S401: Align the posterior latent variables and the interactive features of text and vision F p Concatenate and input into the linear layer for dimensionality reduction;

[0049] Step S402, input the result after dimensionality reduction in step S401 into the masked multi-head attention module of the generator and then perform a residual normalization layer operation;

[0050] Step S403: The output of step S402 and the encoding feature are combined into V N As input to the decoder’s cross-attention module, the fused visual-text feature X is learned. N ;

[0051] Step S404: convert the visual text feature X N ; The dimension is from d mode l becomes V, where V represents the number of words;

[0052] Step S405: at each time step t, based on the classification step p θ (x t |x ＜t ,s ≤t ,I) Select the word x with the highest probability from the vocabulary t Describe the image.

[0053] Compared with the prior art, the present invention has the following advantages: (1) The present invention incorporates image semantic segmentation features to provide guidance for the diversified image description decoder, and effectively visually decouples the target detection features to identify more accurate visual features; (2) The conditional variational Transformer is introduced to learn the one-to-many mapping relationship between image and text by optimizing the variational lower bound objective function, thereby achieving diversified image description generation. A large number of experimental results have shown that the present method is significantly superior to the current mainstream diversified image description model in terms of accuracy and diversity indicators.

[0054] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the mask feature extractor architecture.

[0056] Figure 2 This is a detailed schematic diagram of the model architecture.

[0057] Figure 3 Schematic diagram of the 2-gram diversity of each word position under 20 samples of ISSG-DIC.

[0058] Figure 4 Schematic diagram of the 4-gram diversity of each word position in ISSG-DIC under 20 samples.

[0059] Figure 5 Schematic diagram describing the qualitative comparison between CVT and ISSG-DIC generation.

[0060] Figure 6 A visualization diagram of the attention paid to the image semantic segmentation target during the ISSG-DIC description generation process.

[0061] Figure 7 Illustration of a qualitative comparison of descriptions generated for different image captioning models. DETAILED DESCRIPTION

[0062] Combination Figure 1-Figure 2 A diversified image description generation method based on image semantic segmentation guidance, comprising:

[0063] Step S100, for a given input image I, obtaining image features F, image semantic segmentation target features M and text description X of the input image I;

[0064] Reference Figure 2 Step S200: Use the pre-trained Faster R-CNN and mask feature extractor to process the target detection feature F and the image semantic segmentation target feature M to obtain the final encoding feature F of the input image I. N 、MN , and will be combined into V N ={F N ,M N};

[0065] Step S300, based on the posterior inference sub-network and the prior approximation sub-network in the conditional variational decoding network, the encoding feature combination V N Learn with the text description X to obtain the posterior latent variables of the text Prior hidden variables Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q ;

[0066] Step S400, based on the generator according to the hidden variable s q , interactive features between text and vision F q , combination V N Get an image description.

[0067] Combination Figure 1-Figure 2 The specific process of step S100 includes:

[0068] Step S101, obtaining image features F of image I through a pixel encoder;

[0069] Step S102, obtaining pixel features of the image feature F through a pixel decoder;

[0070] Step S103, defining a cross attention mechanism of a pixel decoder through a kMax decoder, and the pixel decoder obtains a mask embedding vector;

[0071] Step S104, pixel features are multiplied by the mask embedding vector to obtain predicted masks S;

[0072] Step S105, by performing a mask-based pooling operation on the image feature F and the predicted mask S, the image semantic segmentation target feature M is obtained, where H and W are the height and width of the image I, N is the number of segmented regions, h and w are the height and width of the image feature F, D M The feature dimension representing the features of each mask region.

[0073] In step S105, for each region of the image I, under the guidance of the prediction mask S, the image semantic segmentation target feature M is calculated by pooling the image feature F

[0074] M(N,DM )=Σ hw S(N,h,w)·F(h,w,D M ) (1)

[0075] Where S = {s1,s2,...,s N},s i ∈{0,1} H×W Indicates whether each pixel belongs to the i-th region. If s i =1, it means it belongs to the i-th region, otherwise it does not belong to it; N is the number of segmented regions, h and w are the height and width of the image feature F, respectively, D M The feature dimension representing each mask region feature. For convenience, the above process can be expressed as follows:

[0076] M=MaskNet(I) (01)

[0077] in, Afterwards, this method sends the extracted mask feature M to the standard Transformer encoder for encoding, and then introduces the decoding stage to fuse it with the text for decoding.

[0078] Combination Figure 2 , step S200 includes:

[0079] Step S201: for a given input image, use the pre-trained Faster R-CNN and Mask Feature Extractor to extract target detection features and mask region features, respectively. Where R and N represent the number of target features and mask features, respectively. F and D M Represents the respective feature dimensions.

[0080] The image target feature F and the image semantic segmentation target feature M are input into their respective linear layers, and the feature dimension is mapped to d model , and get the feature F 0 、M 0 ;

[0081] Step S202: feature F 0 、M 0 Input into the standard Transformer encoder to obtain the final encoding feature F N 、M N ;in

[0082] The output of the nth layer in the Transformer encoder is input into the multi-head attention module of the n+1th layer for residual normalization

[0083]

[0084]

[0085] Residual normalization followed by a feed-forward network and another residual normalization

[0086]

[0087]

[0088] Where AddNorm(·) represents the residual normalization operation.

[0089] Then, after N layers of iteration, the final encoding feature F of the input image is obtained. N 、M N Finally, we concatenate them to get the combined representation V of the final target feature and the mask feature. N ={F N ,M N}. Afterwards, it is fed into subsequent modules.

[0090] For an input image I, in the diverse image description task, the goal is to generate multiple description sentences X k ,k∈{1,2,…,K}, each description X={x1,x2,…,x T} A sequence of words x of length T t ,t∈{1,2,…,T}. In order to learn the one-to-many mapping from image space to sequence space, a conditional generation model is introduced to maximize the conditional distribution p θ (x|I), where θ is the model parameter. This method is inspired by Seq-CVAE and introduces a sequenced latent variable s=(s1,s2,…,s T ) and input it into the Transformer model, in this way capturing cross-modal information in the image-to-text mapping process. This method is achieved by decomposing the time steps, and the conditional distribution p θ (x|I) is expressed as:

[0091]

[0092] A multiplication operation is used for the time step decomposition.

[0093] Based on the above formula, for the conditional distribution p θ The maximum likelihood estimation of (x|I) can obtain the variational lower bound (ELBO) as follows:

[0094]

[0095] The conditional variational decoding network includes a posterior inference sub-network and a priori approximation sub-network;

[0096] The first term of ELBO represents the log-likelihood of the generated text sequence, and the second term represents the sum of the KL divergence between the posterior and the conditional prior at all time steps during the text generation process. The posterior network q conditioned on the image I and its description sequence x φ (s t |s <t ,x,I) essentially captures the latent space representation of each word in the complete sequence. The prior network p θ (s t |s <t ,x <t ,I) with image I and the currently generated description sequence x <t To condition on this, a latent space is learned for each generated word, which is aligned to the latent space in the posterior network via KL divergence, thereby encouraging the latent space learned in the prior to predict unseen parts of the description.

[0097] Use the posterior network q φ (s t |s <t ,x,I), prior network p θ (s t |s <t ,x <t ,I) and the data log likelihood p θ (x t |x <t ,s ≤t ,I) is used to represent the posterior inference subnetwork, the prior approximation subnetwork, and the generator. Most traditional methods use the pre-trained Faster R-CNN model to extract target-level visual features. This method also incorporates panoramic segmentation features as fine-grained visual features for feature encoding, which provides semantic information guidance for the subsequent variational Transformer and generator, and is conducive to the decoupling of confusing features in target detection features. Therefore, this method FGS-CVT generates more accurate and diverse image description sentences. The specific process is as follows:

[0098] Visual feature V after fusing panoptic segmentation mask feature and object detection feature N The posterior inference subnetwork and the prior approximation subnetwork are respectively passed to perform dual-path variational inference. In order to reduce the complexity of calculation, this method converts s <t Replace with s t-1 , so the prior network and the posterior network can be simply represented as p θ (s t |s t-1,x <t ,I) and q φ (s t |s t-1 ,x,I), and conduct training and inference based on this.

[0099] In step S300, the method in the posterior inference sub-network includes:

[0100] Step S301: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ;

[0101] Step S302: Input vector W 0 Input to the multi-head attention module for self-attention encoding and then perform residual normalization to obtain the text feature W q

[0102] W q =AddNorm(MultiHead(W 0 ,W 0 ,W 0 )) (6)

[0103] in

[0104] Step S303: Use a multi-head cross attention module to analyze the text feature W q and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. q

[0105] F q =AddNorm(MultiHead(W q ,V N ,V N )) (7)

[0106] Step S304, wherein At this time F q It represents the fusion of the observed complete sequence text information and image information.

[0107] Handling conditional distributions with feedforward networks Get the posterior latent variable of the word

[0108]

[0109] The mean and standard deviation are generated by two feed-forward networks, namely Then use the re-parameterization technique to get N(s t ;μt ,σ t ) distribution to obtain the posterior latent variable

[0110]

[0111] in, is the hidden variable of the previous word, is the time-dependent mean μ t and standard deviation σ t Multivariate Gaussian distribution of ;

[0112] In the prior approximation subnetwork, since the complete sequence x cannot be generated in advance during the test phase, the posterior network q φ (s t |s t-1 ,x,I) is discarded, and this method only uses the prior network p θ (s t |s t-1 ,x <t ,I) Sampling s during the test t To make the training and testing process consistent, this method uses the text input vector W in the training phase. 0 The masked multi-head self-attention module is passed in so that it only observes the currently generated text. Therefore, the methods in the prior approximation subnetwork include:

[0113] Step S311: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ;

[0114] Step S312: Input vector W 0 Input to the masked multi-head attention module to obtain the text feature W p

[0115] W p =AddNorm(MaskedMultiHead(W 0 ,W 0 ,W 0 )) (9)

[0116] Step S313, wherein The text feature W p Input to the multi-head cross attention module and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. p

[0117] F p=AddNorm(MultiHead(W p ,V N ,V N )) (10)

[0118] Step S314, using the feedforward network to process the conditional distribution Get the prior latent variables of words

[0119]

[0120] Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q .

[0121] The multi-head cross-attention module in step 313 shares the weights with the multi-head cross-attention module in step S303.

[0122] Step S400 includes:

[0123] Step S401: Align the posterior latent variables and the interactive features of text and vision F p Concatenate and input into the linear layer for dimensionality reduction;

[0124] Step S402, input the result after dimensionality reduction in step S401 into the masked multi-head attention module of the generator and then perform a residual normalization layer operation;

[0125] Step S403: The output of step S402 and the encoding feature are combined into V N As input to the decoder’s cross-attention module, the fused visual-text feature X is learned. N ;

[0126] Step S404: convert the visual text feature X N ; The dimension is from d mode l becomes V, where V represents the number of words;

[0127] Step S405: at each time step t, based on the classification step p θ (x t |x ＜t ,s ≤t ,I) Select the word x with the highest probability from the vocabulary t Describe the image.

[0128] Embodiment 1

[0129] This embodiment trains the above model. The ELBO optimization objective is introduced for training. This embodiment also introduces the cross entropy loss (XE) function for model training. The XE loss and KL loss are as follows:

[0130]

[0131] The XE loss is used to evaluate the similarity between the description generated by the model and the real description annotated by humans, and the KL loss minimizes the KL divergence between the posterior distribution and the prior distribution to make the prior distribution close to the posterior distribution. During the training process, this method combines the XE loss with the KL loss and adopts the joint training as follows:

[0132] L=L XE +αL KL

[0133] Among them, α is a hyperparameter that adjusts the XE loss and KL loss. During the training phase, the posterior inference subnetwork acts as a teacher network and helps the prior approximation subnetwork learn the posterior knowledge of the entire sequence according to the KL loss. Therefore, during the test process, since the prior distribution is already aligned with the posterior distribution, it is feasible to generate the corresponding description based on the prior distribution sampling.

[0134] The image description experiment of this example uses the MSCOCO dataset for training and evaluation, adopts the M-RNN partitioning method, which allocates 118,287 images for training, 4,000 for verification, and 1,000 for testing, and uses the same sentence processing method as the previous study. In order to ensure a comprehensive evaluation, the accuracy index and diversity index introduced above are used to evaluate the quality of the description text generated by FSG-CVT.

[0135] For the FSG-CVT model proposed in this embodiment, this method uses Faster R-CNN and kMax-DeepLab in the visual encoder to extract image target detection features and mask features respectively. The feature dimensions D of the two are F , D M In the posterior inference sub-network, the prior approximation sub-network, and the generator, the dimension d of the word embedding vector, visual features, and latent variables is model The number of heads of the multi-head attention module of the encoder and decoder is 8, and the number of layers N is set to 3. In the training phase, this method sets the batch size to 10 and α to 1.0. In addition, this method also uses the Adam optimization algorithm and the learning rate warmup technique to optimize the proposed FSG-CVT, and uses 5×10 -6The learning rate was set to 0.01 and the training was performed for 30 epochs. In the test phase, 20 and 100 sentences were sampled for each image for fair comparison. In addition, the beam search parameters were the same as those of the compared methods, and the beam width was set to 2 and 1, respectively, for accuracy evaluation and diversity evaluation.

[0136] The FSG-CVT model proposed in this embodiment is compared with various mainstream diversified image description generation methods, including Div-BS, AG-CVAE, PoS, DCL-CVAE, Seq-CVAE and COS-CVAE. Among them, Div-BS is an extension of beam search; AG-CVAE enhances description diversity by introducing additive Gaussian prior in latent space; PoS uses part-of-speech tagging to guide the description generation process; DCL-CVAE uses two-stream conditional variational autoencoder and contrastive learning to enhance the modeling ability of latent space; Seq-CVAE learns the hidden representation of each word based on LSTM framework; COS-CVAE extends the Seq-CVAE model and uses the description of similar images as pseudo-labels to enhance the contextual information in the original dataset.

[0137] The following table compares the accuracy of the method proposed in this embodiment with other methods after Oracle re-ranking on the test set divided by "M-RNN" on the MSCOCO dataset. All methods are trained on the training set divided by "M-RNN". Among them, "sample" represents the number of sentences generated for each image sample, and the best result is shown in bold. The Oracle re-ranking method sorts the generated description text according to the highest score of each evaluation indicator and selects the best description. Each evaluation score is taken as the average on the test set. Consistent with other research works, in the experiments in this section, the number of latent variables sampled is 20 and 100. Oracle re-ranking uses the maximum score of each indicator to sort the generated descriptions and select the best description. Each Oracle evaluation score is the average in the test set. Consistent with previous work, the experiments in this section sample 20 and 100 latent variables and input them into the generator to generate the corresponding number of description sentences.

[0138] The results in the table below show that FGS-CVT scores higher than all other compared methods in various accuracy indicators when sampling 100 descriptions. When using 20 descriptions, except for the B@2 and B@3 indicators, which are slightly lower than COS-CVAE, it achieves the best performance in all other indicators. It is worth noting that it achieved the best results on CIDEr, scoring 168.9 and 199.4 points at sampling 20 and 100, respectively, surpassing the COS-CVAE method by 6.5 and 10.1 points, respectively, which is better correlated with human evaluation. This shows that FGS-CVT can effectively use the mask features of panoptic segmentation to improve the accuracy of description sentences.

[0139] Comparison of the accuracy of the MSCOCO dataset "M-RNN" test set after Oracle re-ranking

[0140]

[0141] The following table compares the diversity results of the proposed method and other methods on the test set partitioned by "M-RNN" on the MSCOCO dataset after consensus rearrangement. For each method, the diversity scores of the top 5 sentences after consensus rearrangement are reported. In the consensus rearrangement, the real descriptions corresponding to the training images similar to the test images are first obtained, and then the generated descriptions are sorted according to the calculated CIDEr scores, and the top 5 sentences are selected.

[0142] As shown in the table below, Div-BS has the highest Unique score, but it is more likely to generate repeated n-gram descriptions, resulting in poor performance on the remaining diversity indicators. In contrast, the FSG-CVT proposed in this method obtained the best Div-1, Div-2, and mBLEU-4 indicators when sampling 20 sentences, and the best Div-1 and Div-2 indicators when sampling 100 sentences, indicating that it can focus on more fine-grained areas in the image, can generate more unique 1-grams and 2-grams, and achieve higher overall performance in all comprehensive indicators. This shows that FSG-CVT can effectively utilize panoramic segmentation features and conditional variational Transformer, learn a more discriminative one-to-many mapping from image to sentence, and generate more diverse description sentences.

[0143] Comparison of diversity results after Consensus re-ranking in the MSCOCO dataset "M-RNN" test set

[0144]

[0145] Reference Figure 3 , Figure 4 In addition, in order to better evaluate the diversity of this method, the two figures are respectively the statistics of the number of unique 2-grams and 4-grams at each word position in the 20 description sentences sampled for each picture in the test set by FSG-CVT and other methods. It can be seen that FSG-CVT generates the most 2-grams and 4-grams for most word positions, respectively. In addition, it can generate unique 2-grams until position 14, which shows that it tends to generate longer sentences compared with other methods. These improvements can be attributed to the unique advantages of FSG-CVT in utilizing the fine-grained semantic guidance of panoramic segmentation features and the conditional variational Transformer in modeling latent variables of words at different positions.

[0146] In this example, the effect of the hyperparameter α on the description performance is first analyzed. Since α is a trade-off factor between the cross entropy loss and the KL loss, it essentially plays a role in balancing the diversity and accuracy of the description sentences. Since α is a trade-off factor between the cross entropy loss and the KL loss, it essentially plays a role in balancing the diversity and accuracy of the description sentences. The following table reports the performance of the model FSG-CVT proposed in this chapter with different α sizes on the accuracy and diversity indicators when sampling 20 description sentences. It can be observed from the table that a larger α can make the model produce more accurate but less diverse descriptions. This is because the KL loss with a larger α value prompts the model to match the prior conditional distribution with the posterior conditional distribution, which can effectively improve the accuracy of the generated description. When α is set to 1.0, diversity and accuracy can be well balanced.

[0147] Performance comparison of FSG-CVT on 20 samples of the MSCOCO dataset "M-RNN" test set under different hyperparameter α settings

[0148]

[0149] Since the setting of different α values ​​verifies the impact of the introduction of the conditional variational transformer on the model performance, this section will verify the impact of the integration of panoramic segmentation features on the model. The following two tables show the impact of panoramic segmentation features on model performance in terms of accuracy and diversity, where CVT means that only the target features extracted by Faster R-CNN are used for visual encoding without using panoramic segmentation features. As can be seen from the table below, FSG-CVT, with the semantic guidance of panoramic segmentation features, has achieved significant improvements in various accuracy indicators, especially when the number of samples is 20 and 100, the CIDEr index has achieved improvements of 4.9 and 1.6 points respectively.

[0150] Ablation experiment on the accuracy of the MSCOCO dataset "M-RNN" test set after Oracle re-ranking

[0151]

[0152]

[0153] At the same time, according to the table below, we can observe that the sentences generated by FSG-CVT have higher diversity indicators than those generated by CVT. The improvements in accuracy and diversity indicators brought by FSG-CVT indicate that panoptic segmentation features can provide fine-grained semantic guidance for image description generation models, which is conducive to generating more accurate and diverse descriptions.

[0154] Ablation experiment on diversity after Consensus re-ranking in the MSCOCO dataset “M-RNN” test set

[0155]

[0156] Reference Figure 5 Two sets of descriptions generated by CVT and FSG-CVT are shown, and the panoramic segmentation map and the true description of the image are given, where the overly general words or phrases in the sentence are indicated in blue, the wrong words are indicated in red, and the fine-grained and discriminative phrases are indicated in green. It can be observed that FSG-CVT can generate more distinctive phrases, such as "a good-looking man" and "white plate". In addition, the sentences it generates also contain more comprehensive background information, such as "in the middle of the room" and "in front of a person". In addition, as shown in the second row, CVT using only object detection features generates the wrong word "fork". This is due to the visual confusion of "plate" and "fork" during training, which leads to a false association between them, resulting in the non-existent wrong word "fork". FSG-CVT integrates the characteristics of panoramic segmentation and can avoid this phenomenon. This shows that the fusion of panoptic segmentation features effectively solves the visual confusion problem in object detection, provides richer global visual information, and can generate more precise and comprehensive sentences.

[0157] Reference Figure 6 ,In order to further qualitatively evaluate the effectiveness of the model’s attention mechanism, Figure 6The attention weight visualization of the panoramic segmentation mask features of the FSG-CVT proposed in this chapter during the description generation process is given. The weight results come from the last head of the cross-attention module in the last layer of the decoder. The mask area with the highest attention weight is marked in red. It can be found that FSG-CVT can pay attention to the relevant areas of the segmented image in most cases. Especially when generating nouns, such as "person", "bicycle", "bus", etc., the attention is successfully focused on the correct mask area, which shows that these mask features provide a clear outline that prompts the model to generate the correct words.

[0158] Reference Figure 7 ,In order to further demonstrate the effectiveness of FSG-CVT on generating diverse ,description sentences, more examples of description generated by FSG-CVT are shown. Figure 7 Demonstrates some descriptions generated by FSG-CVT and other models for test images, where common words and phrases are marked in blue, wrong parts are marked in red, and fine-grained phrases generated by FSG-CVT are marked in green. It can be seen intuitively that this method can generate more accurate and diverse fine-grained descriptions compared with other methods. For example, FSG-CVT accurately identified the color of the cat in the first picture and the number of birds in the second picture, while other methods generated incorrect numbers and words. In addition, other methods tend to generate high-frequency common words and phrases, while FSG-CVT generates more natural and discriminative descriptions, which contain more fine-grained words or phrases such as "wet", "long legs", "brown and white", and "staring".

Claims

1. A diversified image description generation method based on image semantic segmentation guidance, characterized in that: include: Step S100, for a given input image I, obtaining an image feature F, an image semantic segmentation target feature M, and a text description X of the input image I; Step S200: Use the pre-trained Faster R-CNN and mask feature extractor to process the target detection feature F and the image semantic segmentation target feature M to obtain the final encoding feature F of the input image I. N 、M N , and will be combined into V N ={F N ,M N }; Step S300, based on the posterior inference sub-network and the prior approximation sub-network in the conditional variational decoding network, the encoding feature combination V N Learn with the text description X to obtain the posterior latent variables of the text Prior hidden variables Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q ; Step S400, based on the generator according to the hidden variables Interactive features between text and vision F q , combination V N Gets an image description.

2. The method according to claim 1, characterized in that: The specific process of step S100 includes: Step S101, obtaining an image feature F of an image I through a pixel encoder; Step S102, obtaining pixel features of image feature F through a pixel decoder; Step S103, defining a cross attention mechanism of a pixel decoder through a kMax decoder, and the pixel decoder obtains a mask embedding vector; Step S104, multiplying the pixel feature and the mask embedding vector to obtain a predicted mask S; Step S105, by performing a mask-based pooling operation on the image feature F and the predicted mask S, the image semantic segmentation target feature M is obtained, where H and W are the height and width of the image I, N is the number of segmented regions, h and w are the height and width of the image feature F, D M The feature dimension representing the features of each mask region.

3. The method according to claim 2, characterized in that In step S105, for each region of the image I, under the guidance of the prediction mask S, the image semantic segmentation target feature M is calculated by pooling the image feature F M(N,D M )=∑ hw S(N,h,w)·F(h,w,D M ) (1) Where S = {s1,s2,...,s N },s i ∈{0,1} H×W Indicates whether each pixel belongs to the i-th region. If s i =1, it means it belongs to the i-th region, otherwise it does not belong to it; H and W are the height and width of image I, N is the number of segmented regions, h and w are the height and width of image feature F, D M The feature dimension representing the features of each mask region.

4. The method according to claim 1, characterized in that: Step S200 includes: Step S201: input the image target feature F and the image semantic segmentation target feature M into their respective linear layers, and map the feature dimension to d model , and get the feature F 0 、M 0 ; Step S202: feature F 0 、M 0 Input into the standard Transformer encoder to obtain the final encoding feature F N 、M N ;in The output of the nth layer in the Transformer encoder is input into the multi-head attention module of the n+1th layer for residual normalization Residual normalization followed by a feed-forward network and another residual normalization Where AddNorm(·) represents the residual normalization operation.

5. The method according to claim 4, characterized in that The conditional variational decoding network in step S300 includes a posterior inference sub-network and a priori approximation sub-network; in Methods in the posterior inference subnetwork include: Step S301: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ; Step S302: Input vector W 0 Input to the multi-head attention module for self-attention encoding and then perform residual normalization to obtain the text feature W q IN q =AddNorm(MultiHead(In 0 ,IN 0 ,IN 0 )) (6) Step S303: Use a multi-head cross attention module to analyze the text feature W q and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. q F q =AddNorm(MultiHead(W q ,V N ,V N )) (7) Step S304: Processing conditional distribution using feedforward network Get the posterior latent variable of the word in, is the hidden variable of the previous word, is the time-dependent mean μ t and standard deviation σ t Multivariate Gaussian distribution of ; The methods in the prior approximate subnetwork include: Step S311: The text description X embeds each word into a feature vector through an embedding layer and adds the sine code at the corresponding position to obtain an input vector W. 0 ; Step S312: Input vector W 0 Input to the masked multi-head attention module to obtain the text feature W p IN p =AddNorm(MaskedMultiHead(In 0 ,IN 0 ,IN 0 )) (9) Step S313: convert the text feature W p Input to the multi-head cross attention module and the encoding feature combination V N After fusion, the interaction feature F between text and vision is obtained through the residual normalization layer. p F p =AddNorm(MultiHead(W p ,V N ,V N )) (10) Step S314, using the feedforward network to process the conditional distribution Get the prior latent variables of words 6. The method according to claim 5, characterized in that Use KL divergence to transform the posterior latent variable and the prior latent variables Align and obtain the latent variable s q .

7. The method according to claim 5, characterized in that The multi-head cross-attention module in step 313 shares the weights with the multi-head cross-attention module in step S303.

8. The method according to claim 5 or 6, characterized in that: Step S400 includes: Step S401: Align the posterior latent variables and the interactive features of text and vision F p Concatenate and input into the linear layer for dimensionality reduction; Step S402, input the result after dimensionality reduction in step S401 into the masked multi-head attention module of the generator and then perform a residual normalization layer operation; Step S403: The output of step S402 and the encoding feature are combined into V N As input to the decoder’s cross-attention module, the fused visual-text feature X is learned. N ; Step S404: convert the visual text feature X N ; The dimension is from d mode l becomes V, where V represents the number of words; Step S405: at each time step t, based on the classification step p θ (x t |x ＜t ,s ≤t ,I) Select the word x with the highest probability from the vocabulary t Describe the image.

Images