Controllable image description generation method based on part-of-speech guided diffusion network

Abstract

The invention provides a controllable image description generation method based on a part-of-speech guided diffusion network (POSCD-Net), and the method comprises the steps: obtaining image features through a pre-trained CLIP model, and enabling the image features to be projected into a potential embedding space through a trained mapping network MapNet; a word sequence generator based on diffusion processes the tag sequence to obtain word vectors, predicts noise, and reconstructs the tag sequence according to the predicted noise; and a diffusion-based descriptive generator processes the word vectors, the reconstructed tag sequence and the image features to obtain the description of the image.

Description

Technical Field

[0001] The present invention relates to an image description generation technology, in particular to a controllable image description generation method based on a part-of-speech guided diffusion network. Background Art

[0002] Image caption generation is a fundamental task in the field of vision and language, which aims to describe the relevant semantics of an image in natural language sentences. With the advanced Transformer-based encoder-decoder architecture, current autoregressive caption generation models have achieved significant progress on most accuracy-based evaluation metrics. Diffusion models improve the accuracy of image caption generation in a non-autoregressive manner. However, these accuracy-oriented models often tend to learn patterns that frequently appear in the training corpus, which is the so-called model collapse problem. To address the model collapse problem, researchers mainly focus on two aspects: (1) diverse image caption generation (DIC), that is, improving the diversity of generated captions; (2) controllable image caption generation (CIC), that is, enhancing the controllability of image captions. In terms of DIC, several approaches have been proposed, such as generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models (DMs). These diffusion-based DIC methods still face two limitations: (1) like traditional diffusion models, the quality of generated samples fluctuates; (2) there is limited control over the properties of generated sentences. In the CIC field, various methods have been proposed to actively manipulate the content or structure of generated sentences, but they are still limited in two aspects: (1) existing models, whether relying on object regions, length-level attributes, or part-of-speech (POS) tags, cannot generate sentences that are both accurate and diverse; (2) current fine-grained CIC models that leverage scene graphs, representative patterns, and the semantic roles of specific verbs cannot precisely control the number of words in the target sentence and the corresponding number of POS tags. Summary of the invention

[0003] The object of the present invention is to provide a controllable image description generation method based on a part-of-speech guided diffusion network (POSCD-Net), comprising:

[0004] Step S100, using the pre-trained CLIP model to obtain the image feature I, and projecting the image feature I into the potential embedding space through the trained mapping network MapNet;

[0005] Step S200, a diffusion-based word sequence generator obtains word vectors by processing the tag sequence Prediction Noise According to the predicted noise Reconstruct the label sequence;

[0006] Step S300: Diffusion-based descriptive generator for word vectors Reconstruct the label sequence and image feature I to obtain the description of the image.

[0007] Furthermore, in step S100, the pre-trained CLIP model is used to obtain the image feature I, and the trained mapping network MapNet projects the image feature I into the potential embedding space to obtain V

[0008] I=CLIP(image)

[0009]

[0010] in Indicates visual prefix, P img and T img represent position embedding and category embedding respectively.

[0011] Further, step S200 includes:

[0012] Step S201: embed a tag sequence P through an embedding function Emb pos (*) Mapped into a vector

[0013] Step S202, from the distribution Sampling word vectors Among them, Emb pos represents the POS embedding layer with parameter φ2, α0 is the amount of noise added in the initial step, and I represents the identity matrix;

[0014] Step S203: According to formula (2), Gradually add noise α t , until time step T is obtained

[0015]

[0016] in, The parameter ε follows a normal distribution, Through the upsampling network, The dimension is aligned with the image feature dimension;

[0017] Step S204: With position embedding P pos and category embedding T pos Add to get

[0018]

[0019] Step S205: and V pos Merged into a sequence In BERTlayer, according to the posterior probability right Denoising is performed to obtain And split it up and use As the downsampled input, the predicted output is obtained

[0020] Step S206: output according to the prediction Reconstruct the label sequence P'.

[0021] Further, step S300 includes:

[0022] Step S301: Embed the learnable vector to obtain the conditional input u;

[0023] Step S302: The reconstructed tag sequence P' is embedded in the pos (*) Mapped into a vector;

[0024] Step S303, from the distribution Sampling word vectors φ1 represents Emb cap (*) is a trainable parameter, α'0 represents the amount of noise added in the initial step;

[0025] Step S304: Gradually increase the noise to get

[0026]

[0027] Step S305: Split the output of each BERT layer into visual features and noisy data in Depend on With position embedding P cap and category embedding T cap Add to obtain;

[0028] Step S306, calculate the cross attention of the noise data and the conditional input u, and obtain

[0029]

[0030] Step S307, and Merge into a sequence and use the posterior probability in the BERT Layer Prediction Output according to Get image description.

[0031] Further, in step S306, the conditional input u is mapped into a key (K) and a value (V) in the Syntactic Control Module after downsampling. Mapped to query (Q), obtained through multi-head attention, normalized residual, feedforward network, and normalized residual

[0032]

[0033] The present invention applies continuous diffusion to the task of controllable image description generation, and has the following advantages over the prior art: (1) it seamlessly integrates part-of-speech (POS) sequence prediction, classifier-free guidance, and syntactic control module (SCM) to improve the controllability of the diffusion-based image description generation process, thereby alleviating the problems of word duplication and omission in the traditional diffusion process; (2) extensive experiments are carried out on the MSCOCO dataset, and the results show that the model provided by the present invention outperforms the most advanced non-autoregressive models in performance and is comparable to the autoregressive models, while having higher controllability.

[0034] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0036] Figure 2 Schematic diagram of the DPG and DCG architecture.

[0037] Figure 3 A qualitative comparison of descriptions generated by different image caption generation models.

[0038] Figure 4 Example graph of diverse part-of-speech sequences and corresponding image descriptions generated by the POSCD-Net model.

[0039] Figure 5 An example graph of words generated by manually input POS tags. DETAILED DESCRIPTION

[0040] Combination Figure 1 , a controllable image description generation method based on part-of-speech guided diffusion network (POSCD-Net), comprising:

[0041] Step S100, using the pre-trained CLIP model to obtain the image feature I, and projecting the image feature I into the potential embedding space through the trained mapping network MapNet;

[0042] Step S200, a diffusion-based word sequence generator obtains word vectors by processing the tag sequence Prediction Noise According to the predicted noise Reconstruct label sequence;

[0043] Step S300: Diffusion-based descriptive generator for word vectors Reconstruct the label sequence and image feature I to obtain the description of the image.

[0044] In step S100, the pre-trained CLIP model (Contrastive Language-Image Pre-Training) is used to obtain the image feature I, and the trained mapping network MapNet projects the image feature I into the potential embedding space.

[0045]

[0046] in Indicates visual prefix, P img and T img Denote position embedding and category embedding respectively. In the subsequent steps, the diffusion-based POS Generator (DPG) and the diffusion-based caption generator (DCG) share V, which is represented in DPG as In DCG, it is represented as Where V pos and V cap All belong to m represents the number of features, and d2 represents the dimension of each image feature.

[0047] The optimization objective (loss function) of DPG in step S200 is:

[0048]

[0049] Among them, P represents a label sequence of POS, Used to optimize the entire diffusion model, The embedding layer used to train POS, For use from Get P; the three loss functions are

[0050]

[0051] f θ Represents the denoising model in DPG, regular term Is Computational The purpose is to prevent the norm of the embedding layer from being too large.

[0052] According to the optimization goal, the DPG process is as follows:

[0053] Step S201: embed a tag sequence P through an embedding function Emb pos (*) maps to a Vector

[0054] Step S202, from the distribution Sampling word vectors Among them, Emb pos represents a POS embedding layer with parameter φ2, And k is the number of POS tags, d3 is the dimension of POS embedding, α0 is the amount of noise added in the initial step, and I represents the identity matrix;

[0055] Step S203: According to formula (2), Gradually add noise α t , until time step T is obtained

[0056]

[0057] in, The parameter ε follows a normal distribution, Through the upsampling network, The dimension is aligned with the image feature dimension;

[0058] Step S204: With position embedding P pos and category embedding T pos Add to get

[0059]

[0060] Step S205: and V pos Merged into a sequence In BERTlayer, according to the posterior probability right Denoising is performed to obtain And split it up and use As the downsampled input, the predicted output is obtained

[0061] Step S206: output according to the prediction Reconstruct the label sequence P'.

[0062] In step S203, at time step T, Approximates pure Gaussian noise.

[0063] In step S205, the posterior probability is a mean value Gaussian distribution with variance σ(t), that is

[0064]

[0065] According to formula (4)(5)(6), it can be To predict

[0066] In step S300, the optimization objective (loss function) of DCG is:

[0067]

[0068] Among them, C represents a description statement, Forced diffusion model based on predict For training, map C to The embedding layer; Represents the data likelihood, used to predict Reconstruct C in the above example. The three loss functions are

[0069]

[0070] Among them, g θ Represents the denoising model in DCG.

[0071] According to the optimization target, the DCG processing process in step S300 includes:

[0072] Step S301: Embed learnable parameters to obtain the conditional input u of DCG;

[0073] Step S302: The reconstructed tag sequence P' is embedded in the pos (*) maps to a A vector of

[0074] Step S303, from the distribution Sampling word vectors φ1 represents Emb cap (*) Trainable parameters, k is the number of labels in the reconstructed label sequence, and α'0 represents the amount of noise added in the initial step;

[0075] Step S304: Gradually increase the noise to get

[0076]

[0077] Step S305: Split the output of each BERT layer into two parts: visual features and noisy data in Depend on With position embedding P cap and category embedding T cap Add to obtain;

[0078] Step S306, calculating the cross attention between the noise data and the conditional input u in the Syntactic Control Module (SCM);

[0079] Step S307, and Merge into a sequence and use the posterior probability in the BERT Layer Prediction Output according to Get image description.

[0080] In step S306, SCM incorporates POS information into the diffusion process of description, which can improve the controllability of DCG. In SCM, the conditional input u is mapped to key (K) and value (V) after downsampling. Mapped to query (Q), obtained through multi-head attention (Multi-HeadAttention), normalized residual (Add&Norm, Feed Forward), feedforward network (FeedForward), normalized residual (Add&Norm, Feed Forward)

[0081] Comparative Example

[0082] This example compares POSCD-Net on the widely used offline MS COCO m-RNN segmentation dataset. The MS COCO dataset contains more than 93,000 images, each with five manually annotated descriptions. In addition, this example follows the same sentence preprocessing settings in previous work, which include:

[0083] Convert all words in the description to lowercase to avoid duplication due to different capitalizations.

[0084] Eliminate words that appear less than five times to reduce the impact of rare words on model performance;

[0085] Remove some misspelled words, which will not significantly affect the diversity of the vocabulary.

[0086] In addition, the length of each sentence is set to 30, and for shorter sentences, padding is performed by adding [PAD] tokens. These preprocessing steps ensure the consistency of the experiments.

[0087] 1. Implementation details

[0088] In this embodiment, the image, part-of-speech tagging and sentence features are all 512-dimensional after upsampling. The visual encoder uses a pre-trained CLIP image encoder based on ViT-B / 32. In order to align with the visual and text feature dimensions, the denoising model uses a 512-dimensional BERT, whose encoding layer contains multi-head attention, with eight attention heads per layer. The Stanford Log-linear Part-of-Speech (POS) Tagger is used to extract fine-grained POS tags for each real sentence. These tags are divided into twelve categories: verbs (VERB), nouns (NOUN), adjectives (ADJ), adverbs (ADV), conjunctions (CONJ), pronouns (PRON), prepositions (ADP), particles (PRT), numeral determiners (NUM), determiners and articles (DET), others (X), and phrase end markers (EOP).

[0089] During the training phase, the dimension of POS embedding was set to 48, and DPG used the BERT architecture with 12 layers. The noise schedules of DCG and DPG were both truncated cosine functions. The AdamW optimizer was used, and the learning rate was set to 1×10 -4 In the inference stage, a hybrid approach is adopted to combine the POS sequences of the five sentences in the test set with the POS sequences we generated, which can effectively guide the sampling process.

[0090] 2. Evaluation Criteria

[0091] Two standard metrics were used to comprehensively assess the diversity and quality of descriptions.

[0092] (1)Oracle Best-1 Accuracy

[0093] Best-1 accuracy, that is, for each evaluation metric, select the candidate description with the highest score. The oracle accuracy metrics used include:

[0094] BLEU: used for precision-based evaluation, mainly measuring the accuracy of machine translation.

[0095] METEOR: Considers both precision and recall, and evaluates the quality of translation or description through exact matching, synonym matching, stem matching, etc.

[0096] ROUGE-L: An overlap-based similarity measure that evaluates the similarity between two sentences by the longest common subsequence (LCS).

[0097] CIDEr: evaluates the accuracy of image descriptions by n-gram similarity, with special emphasis on the role of rare n-grams.

[0098] SPICE: It is used to evaluate the semantic similarity between sentences. It uses scene graphs to represent the semantic information in sentences and calculates the similarity between scene graphs.

[0099] These metrics provide a comprehensive assessment of the quality of the generated descriptions, covering multiple aspects from the lexical level to the semantic level.

[0100] (2)Diversity

[0101] The diversity metrics used are as follows:

[0102] Uniqueness: represents the proportion of unique descriptions generated during the test phase. This metric measures the novelty and non-repetitiveness of the descriptions generated by the model.

[0103] Novel sentences: refers to descriptions that have never appeared in the training set. This metric evaluates the model's ability to generate unseen, novel sentences.

[0104] mBleu-4: For each description generated for each image, its Bleu-4 score is calculated, and then the Bleu-4 scores are averaged over the entire test set. Higher mBleu-4 values ​​indicate lower diversity, as high Bleu-4 scores generally mean that descriptions are similar to those in the training data.

[0105] Div-n: Diversity is measured by calculating the proportion of different n-grams in a set of generated diverse descriptions. A higher Div-n value indicates that the description contains more different n-gram combinations, thus having higher diversity.

[0106] Self-CIDEr: refers to the CIDEr kernelized diversity metric, which evaluates diversity by computing the singular value decomposition (SVD) of the autocorrelation matrix of diverse descriptions. Self-CIDEr is able to capture the semantic differences between descriptions, thus more accurately evaluating the diversity of generated descriptions.

[0107] 3. Quantitative analysis

[0108] The Best-1 accuracy of the POSCD-Net method and strong baselines on the m-RNN test set is evaluated following an oracle re-ranking process. The oracle re-ranking process selects the description with the highest score for each metric as the Best-1 description. These oracle evaluation scores are then averaged over the entire test set. Table 1 shows the Best-1 accuracy metrics for autoregressive and non-autoregressive methods for 20 and 100 samples. These results show the accuracy and diversity of different methods in generating image descriptions.

[0109] Table 1

[0110]

[0111] B@1:BLEU-1, B@2:BLEU-2, B@3:BLEU-3, B@4:BLEU-4, M:METEOR, R:ROUGE-L, C:CIDEr, andS:SPICE.

[0112] Table 1. Comparison of the m-rnn test split with the oracle performance baseline (i.e., Best-1 quality) on MS COCO. “c20” and “c100” represent 20 and 100 samples per image, respectively, with the best results in bold. Higher scores indicate higher overall accuracy.

[0113] (1) Accuracy assessment

[0114] The Best-1 accuracy of the POSCD-Net method and strong baseline methods are evaluated on the m-RNN test set, following oracle reranking. The oracle reranking process selects the description with the highest score for each metric as the Best-1 description. Then, the evaluation scores of the oracle are averaged over the entire test set. Table 1 reports the Best-1 accuracy metrics of the autoregressive and non-autoregressive methods for 20 and 100 samples. It can be seen that POSCD-Net achieves higher accuracy than the prefix diffusion model in the same non-autoregressive manner. This is because the POSCD-Net method seamlessly combines the SCM module with classifier-free guidance to further improve the controllability of the continuous diffusion model. Since there are few non-autoregressive methods to generate diverse image descriptions before, the POSCD-Net method is mainly compared with the state-of-the-art autoregressive methods. From Table 1, it can be seen that in the case of 20 samples, POSCD-Net can significantly outperform the autoregressive methods. Specifically, when the guidance strength is set to 1.5, POSCD-Net's CIDEr score is better than Transformer-DML (1.749 vs. 1.704), DCL-CVAE (1.749 vs. 1.502), and UNKT (1.749 vs. 1.742). For 100 samples, the POSCD-Net method outperforms DCL-CVAE and is comparable to Transformer-DML and UNKT. Experimental results on accuracy show that POSCD-Net can effectively improve the performance of traditional continuous diffusion-based image description generation. In particular, it significantly narrows the gap between autoregressive and non-autoregressive methods by introducing POS control signals into the diffusion-based language model.

[0115] Table 2

[0116]

[0117] Table 2 shows the diversity evaluation of the best 5 sentences after re-ranking according to the diversity index of the m-rnn test. The symbols “↑” and “↓” indicate that higher is better and lower is better, respectively.

[0118] (2) Diversity Assessment

[0119] From the results, we can see that POSCD-Net is comparable to the non-autoregressive prefix diffusion model in the case of 20 and 100 samples. However, the prefix diffusion model cannot control the description generation by external signals. Compared with the autoregressive method, the POSCD-Net method scores lower on some diversity indicators. Nevertheless, the POSCD-Net method can maintain diversity as a whole and even achieve comparable performance on some indicators such as Div-1 and Div-2. Although the POSCD-Net method cannot surpass the autoregressive model in diversity, it performs better in terms of controllability. Compared with pattern-based image description generation (CIC) models such as Transformer-DML and POS, POSCD-Net can provide more fine-grained control for description generation at the word level. If the number of predefined patterns changes, these models must also be retrained. In contrast, POSCD-Net can learn all pattern representations in the training corpus with just one training. Therefore, the POSCD-Net method is more flexible because it is not restricted by predefined patterns. In addition, POSCD-Net is more convenient than existing CIC models.

[0120] 4. Qualitative analysis

[0121] (1) Diversity

[0122] We further compare the diverse image descriptions generated by POSCD-Net with several baseline DIC methods. Figure 3 As shown in Figure 1, baseline methods, while able to generate diverse descriptions, often produce inaccurate vocabularies. This is because these baseline methods are trained to directly map an image to a set of descriptions without external control signals, resulting in poor interpretability and limited generalization performance. In contrast, our method is able to generate more accurate descriptions as it utilizes a two-stream diffusion architecture to decouple syntax and semantics to improve the controllability of description generation. For the second image, although POSCD-Net still produces high-frequency n-grams due to its non-autoregressive nature, it achieves a better balance between diversity and accuracy than other methods.

[0123] (2) Controllability

[0124] In order to evaluate the controllability of POSCD-Net, Figure 4Three test images and their corresponding descriptions guided by the POS sequences generated by DPG are shown in Figure 2. It is observed that for these test images, most of the predicted vocabularies are consistent with the corresponding generated POS tags, which indicates that POSCD-Net has superior generalization performance during the test phase. In addition, it is worth noting that with the help of the NUM tag, POSCD-Net is able to accurately count the number of objects of the same kind in the first and third images. For the third image, POSCD-Net is able to generate diverse and appropriate phrases such as "opengrassy" and "lush green" using the same two ADJ tags, demonstrating its controllability and diversity in generating fine-grained vocabularies.

[0125] In order to further verify the controllability of POSCD-Net, Figure 5 In , we show some images and descriptions generated from manually entered POS tags. Figure 5 In the left column of , when POSCD-Net uses the same POS tag sequence as a control signal to generate descriptions for different images, it is able to generate appropriate vocabulary for each specified POS tag in most cases. For example, when POSCD-Net is provided with a POS tag sequence such as "NUM ADJ NOUN..." for each image, it is able to accurately generate "Two professional players...", "Four brown bears...", and "Four young zebras...". Figure 5 In the right column of , some tags in each POS tag sequence in the left column are deleted, resulting in a set of simplified POS tag sequences. These simplified sequences are used as control signals to test the controllability of POSCD-Net. It can be observed that POSCD-Net is still able to successfully control the generation of the corresponding vocabulary. These experimental results verify that POSCD-Net is able to control the description generation process from coarse to fine. By using POS (part of speech) tags as control signals, POSCD-Net is able to accurately generate vocabulary that matches the given tags, and can generate diverse and appropriate phrases when needed. This not only demonstrates the controllability of the model, but also proves its flexibility and accuracy in handling image description generation tasks.

Claims

1. A controllable image description generation method based on part-of-speech guided diffusion network (POSCD-Net), comprising: Step S100, using the pre-trained CLIP model to obtain the image feature I, and projecting the image feature I into the potential embedding space through the trained mapping network MapNet; Step S200, a diffusion-based word sequence generator obtains word vectors by processing the tag sequence Prediction Noise According to the predicted noise Reconstruct the label sequence; Step S300: Diffusion-based descriptive generator for word vectors Reconstruct the label sequence and image feature I to obtain the description of the image.

2. The method according to claim 1, characterized in that In step S100, the pre-trained CLIP model is used to obtain the image feature I, and the trained mapping network MapNet projects the image feature I into the potential embedding space to obtain V I=CLIP(image) in Indicates visual prefix, P img and T img represent position embedding and category embedding respectively.

3. The method according to claim 2, characterized in that Step S200 includes: Step S201: embed a tag sequence P through an embedding function Emb pos (*) Mapped into a vector Step S202, from the distribution Sampling word vectors Among them, Emb pos represents the POS embedding layer with parameter φ2, α0 is the amount of noise added in the initial step, and I represents the identity matrix; Step S203: According to formula (2), Gradually add noise α t , until time step T is obtained in, The parameter ε follows a normal distribution, Through the upsampling network, The dimension is aligned with the image feature dimension; Step S204: With position embedding P pos and category embedding T pos Add to get Step S205: and V pos Merged into a sequence In the BERT layer, according to the posterior probability right Denoising is performed to obtain And split it up and use As the downsampled input, the predicted output is obtained Step S206: output according to the prediction Reconstruct the label sequence P'.

4. The method according to claim 3, characterized in that Step S300 includes: Step S301: Embed the learnable vector to obtain the conditional input u; Step S302: The reconstructed tag sequence P' is embedded in the pos (*) Mapped into a vector; Step S303, from the distribution Sampling word vectors φ1 represents Emb cap (*) is a trainable parameter, α'0 represents the amount of noise added in the initial step; Step S304: Gradually increase the noise to get Step S305: Split the output of each BERT layer into visual features and noisy data in Depend on With position embedding P cap and category embedding T cap Add to obtain; Step S306, calculate the cross attention of the noise data and the conditional input u, and obtain Step S307, and Merge into a sequence and use the posterior probability in the BERT Layer Prediction Output according to Get image description.

5. The method according to claim 4, characterized in that In step S306, the conditional input u is mapped into a key (K) and a value (V) in the Syntactic ControlModule module after downsampling. Mapped to query (Q), obtained through multi-head attention, normalized residual, feedforward network, and normalized residual

Images