Diversified image description generation method based on cascade self-conditional diffusion network

Abstract

The invention provides a cascaded self-conditional diffusion network-based diversified image description generation method, which is characterized by comprising a plurality of cascaded sub-conditional diffusion modules, the prediction noise of each stage of sub-condition diffusion module is obtained from the visual mark, the word vector and the prediction noise of the previous stage of sub-condition diffusion module according to a training target, and the final output noise is obtained from the prediction noise of the last stage of sub-condition diffusion module according to a trainable language model head; and obtaining image description according to the final output noise.

Description

Technical Field

[0001] The invention relates to an image description generation technology, in particular to a diversified image description generation method based on a cascaded self-conditional diffusion network. Background Art

[0002] Image captioning is a fundamental task in the field of vision and language, which aims to generate natural sentences to describe the key semantics of an image. Diffusion models (DMs) have been successfully applied to image captioning due to their simplified training process, stability, and highly parallelizable non-autoregressive sentence generation capabilities. Unlike traditional autoregressive methods that generate captions token by token, diffusion-based image captioning methods start by sampling random Gaussian noise and iteratively refine it under the guidance of image features to eventually generate the entire description sentence. Traditional diffusion-based image captioning methods usually condition the image embedding as a denoising model, which is trained to recover the original sentence data from the uncorrupted data. However, these methods still face two limitations: (1) the projection layer in the language image contrastive pre-training model (CLIP) leads to the loss of fine-grained visual semantics; (2) the quality of generated samples fluctuates and there is a problem of repeated vocabulary. Summary of the invention

[0003] The present invention provides a diversified image description generation method based on a cascaded self-conditional diffusion network, comprising a plurality of cascaded sub-conditional diffusion modules, wherein each level of sub-conditional diffusion modules obtains the prediction noise of the sub-conditional diffusion module of this level by converting visual tags, word vectors, and the prediction noise of the previous level of sub-conditional diffusion modules according to the training objectives, and the prediction noise of the last level of sub-conditional diffusion modules obtains the final output noise according to the trainable language model head. ,according to Gets the image description.

[0004] Furthermore, each level of the conditional diffusion module includes the following steps:

[0005] Step S100, obtaining Q visual markers of the image feature V, compressing the two-dimensional matrix of the visual markers through a two-dimensional adaptive average pooling operation, transforming the compressed two-dimensional matrix into a one-dimensional array and aligning it with the image embedding to obtain V';

[0006] Step S200, obtaining the predicted noise of the i-th conditional diffusion module according to the training target .

[0007] Furthermore, the specific process of step S100 includes:

[0008] Step S101, obtaining Q visual labels of image features V;

[0009] Step S102: rearrange the Q visual markers into a dimension of A two-dimensional matrix of

[0010] Step S103, using a two-dimensional adaptive average pooling operation (2D AvgPool) to transform the dimension The two-dimensional matrix is ​​compressed to the dimension A collection of compressed tags;

[0011] Step S104, converting the compressed set of compressed markers into a one-dimensional array containing P markers;

[0012] Step S105, embedding the one-dimensional array and the image through linear transformation alignment, as a query for the tag projector, d I is the dimension of image embedding;

[0013] Step S106, the mark projector maps the aligned data to the same space as the aligned image embedding to obtain V'.

[0014] Furthermore, the specific process of step S200 includes:

[0015] Step S201: According to the forward process of formula (3), at any time step t, and Sampling Noise Data

[0016] (3)

[0017] Among them, d i is the i-th sub-conditional diffusion module, is the word vector, Reconstruct the result for the i-1th sub-conditional diffusion module, represents the Gaussian noise associated with time step t, and I is the identity matrix;

[0018] Step S202, embed the time into t emb With noisy data Add element by element;

[0019] Step S203, for time embedding and V', according to the posterior distribution prediction of formula (4)

[0020] (4)

[0021] in

[0022]

[0023]

[0024] in, .

[0025] Furthermore, the prior conditional distribution of formula (5) is reparameterized to train the sampled word vectors

[0026] (5)

[0027] Step by step to word vector Add Gaussian noise

[0028] (6)

[0029] Until the diffusion step T, the sampled Approximately Gaussian distribution.

[0030] Furthermore, we can use the closed form method with random time steps t and sampling , formula (6) is adjusted to

[0031] . (7)

[0032] Compared with the prior art, the present invention has the following advantages: (1) The present invention designs a VTA module to alleviate the problem of visual semantic loss and enhance visual-language alignment; (2) A novel self-conditional diffusion process is introduced to utilize the rich knowledge from previous predictions with high sampling efficiency; (3) Multiple self-conditional diffusion modules are stacked in a cascade manner to gradually improve the accuracy and diversity of image description.

[0033] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0035] Combination Figure 1 , a diversified image description generation method based on cascaded self-conditional diffusion network (CSD-Net) and visual feature alignment, stacking multiple sub-conditional diffusion modules in CSD-Net, and gradually enhancing the diversity and accuracy of the results through fine vision-language alignment and self-conditional denoising. The cascade diffusion process is shown in formula (1)

[0036] (1)

[0037] Where n represents the total number of stacked diffusion modules, d0 is the initial diffusion module, without previous predictions, set This process allows each diffusion module d i ( ) Based on the previous prediction The diffusion process is performed and finally a trainable language model head (LM-head) is used to decode the final output to obtain .according to Gets the image description.

[0038] Before performing each sub-conditional diffusion module operation, the image I and description C are first operated as follows:

[0039] A CLIP-based frozen image encoder extracts features of image I , where Q is the number of visual markers, d V is its dimension.

[0040] Map the description C containing d words to a vector Here, d is the vector dimension. The mapping function w2v(w i ), the embedding of a description C containing d words is defined as

[0041] (2)

[0042] Each child conditional diffusion module performs the following operations:

[0043] Step S100, obtaining Q visual markers of the image feature V, compressing the two-dimensional matrix of the visual markers through a two-dimensional adaptive average pooling operation, transforming the compressed two-dimensional matrix into a one-dimensional array and aligning it with the image embedding to obtain V';

[0044] Step S200, obtaining the predicted noise of the i-th conditional diffusion module according to the training target .

[0045] The specific process of step S100 includes:

[0046] Step S101, obtaining Q visual markers (patches) of the image feature V;

[0047] Step S102: rearrange the Q visual markers into a dimension of A two-dimensional matrix of

[0048] Step S103, using a two-dimensional adaptive average pooling operation (2D AvgPool) to transform the dimension The two-dimensional matrix is ​​compressed to the dimension A collection of compressed tags;

[0049] Step S104, converting the compressed set of compressed markers into a one-dimensional array containing P markers;

[0050] Step S105, embedding the one-dimensional array and the image through linear transformation Alignment, as a query to the Token Projector, d I is the dimension of image embedding;

[0051] In step S106, the marker projector maps the aligned data to the same space as the aligned image embedding to obtain V', thereby enhancing fine-grained image-text alignment during the denoising process.

[0052] In step S100, The image-text similarity is very good, but the information is very little; the two-dimensional adaptive average pooling operation can contain more comprehensive information, but the image-text similarity is poor. Therefore, through the alignment operation in step S106, both good image-text similarity and comprehensive information are achieved.

[0053] The denoising process of step S200 makes the following assumptions:

[0054] (1) The i-th diffusion module d i The forward process transition condition depends on the reconstruction result of the previous diffusion module ,Right now ;

[0055] (2) d i The reverse process transformation depends only on .

[0056] Step S200 performs training according to the following training objectives to obtain the conditional diffusion modules:

[0057]

[0058] The specific process of step S200 includes:

[0059] Step S201: According to the forward process of formula (3), at any time step t, and Sampling Noise Data

[0060] (3)

[0061] Among them, d i is the i-th sub-conditional diffusion module, is the word vector, Reconstruct the result for the i-1th sub-conditional diffusion module, represents the Gaussian noise associated with time step t, and I is the identity matrix;

[0062] Step S202, embed the time into t emb With noisy data Add element by element;

[0063] Step S203, for time embedding and V', according to the posterior distribution prediction of formula (4)

[0064] (4)

[0065] in

[0066]

[0067]

[0068] in, .

[0069] In step S201, the prior conditional distribution of formula (5) is trained using reparameterization to sample word vectors

[0070] (5)

[0071] Step by step to word vector Add Gaussian noise

[0072] (6)

[0073] Until the diffusion step T, the sampled Approximate Gaussian distribution;

[0074] It can be expressed in closed form with random time steps t and sampling

[0075] (7)

[0076] Example

[0077] This example verifies the effectiveness of CSD-Net on the MS COCO dataset, which contains 123,287 images, each with 5 manually annotated sentences. To ensure fairness, this example compares the CSD-Net method with the widely used offline m-RNN segmentation, which contains 118,287 images for training, and 4K and 1K images for validation and testing, respectively.

[0078] 1. Evaluation Metrics

[0079] This example evaluates the best-1 accuracy of the CSD-Net approach and a strong baseline on the m-RNN test split after oracle reranking. Oracle reranking uses the highest score for each metric as the oracle to rank the diverse descriptions and select the best description. Each oracle evaluation score is averaged over the entire test set. Oracle accuracy metrics include BLEU (Bn), CIDEr (C), ROUGE (R), METEOR (M), and SPICE (S). The diversity metrics used are as follows: Uniqueness refers to the proportion of different descriptions generated during the test phase. Novel sentences refer to descriptions that have never been seen in the training set. mBleu-4 calculates the Bleu-4 score of the description generated for each image and then averages the Bleu-4 scores across the test set. Higher mBleu-4 values ​​indicate lower diversity. Div-n measures diversity and is evaluated by calculating the proportion of different n-grams in a set of generated diverse descriptions. Self-CIDEr is a diversity metric kernelized by CIDEr and is calculated by performing a singular value decomposition (SVD) on the autocorrelation matrix of the diverse descriptions.

[0080] 2. Experimental details

[0081] CSD-Net uses 12 attention heads in all Transformer decoder blocks, with a hidden layer size of 768. The size of the feedforward filter is set to 3072, and the dropout rate is 0.1. In terms of visual representation, CSD-Net adopts CLIP ViT-L / 14 as the visual backbone network, and reduces the visual tags from 576 to 144 through 2D AvgPool, with the same settings as previous work. The tag projector is a Transformer decoder with 3 blocks. To ensure fair comparison, the same batch size as other studies is set, and the number of diffusion time steps is set to 1000. The noise schedule of the denoiser uses a truncated cosine function. We use the AdamW optimizer with a learning rate of , and the cosine decay is set to 0.01. CSD-Net is trained for 200,000 iterations on an NVIDIA 4090 GPU. The 10-step inference time is 0.08 seconds per image, and the 15-step inference time is 0.12 seconds per image.

[0082] 3. Overall performance

[0083] (1) Accuracy assessment

[0084] The methods and strong baselines were evaluated on the m-RNN test set using Oracle reranking. Table 1 shows the best-1 accuracy metrics for autoregressive and non-autoregressive methods at 20 and 100 samples. DDIM was used to sample descriptions at different steps (10 and 15). According to the evaluation results, CSD-Net achieved significant improvements over the previous autoregressive DIC model in most metrics. In addition, CSD-Net reproduced Prefix-diffusion on m-RNN segmentation. Using the same network parameters, it was found that CSD-Net significantly outperformed Prefix-diffusion in CIDEr scores at different sampling steps (2.080 vs. 1.988 and 2.074 vs. 1.988). This shows that the VTA and cascaded self-conditional denoiser proposed by CSD-Net can effectively improve the accuracy of diffusion-based models. The CSD-Net model was further trained on MSCOCO Karpathy segmentation and compared with the current diffusion model in terms of accuracy metrics. As shown in Table 2, CSD-Net surpasses most existing diffusion models in all accuracy metrics, indicating that the sampling quality can be effectively improved through visual marker alignment and cascade structure optimization. Although its accuracy is slightly lower than LaDiC, CSD-Net only requires 23% of LaDiC parameters (148 / 637).

[0085] Table 1

[0086]

[0087] Table 1 Best-1 accuracy results obtained for the M-RNN test split on the MSCOCO dataset. “c20” and “c100” indicate sampling 20 and 100 descriptions per image, respectively, and the best results are marked in bold. Higher scores in all metrics indicate higher accuracy.

[0088] Table 2

[0089]

[0090] Table 2. Performance comparison on COCO image caption Karpathy segmentation using diffusion-based models. Params denotes the total number of parameters. The symbols “↑” and “↓” denote that higher scores are better and lower scores are better, respectively.

[0091] (2) Diversity Assessment

[0092] Table 3 evaluates the diversity of different image description models. We show the diversity scores of each method on the best 5 sentences using a consistent reranking process. Compared with Prefix-diffusion, CSD-Net performs better on most diversity evaluation metrics when the sampling step number is 10, demonstrating its effectiveness in DIC. In addition, it can be observed that the diversity of CSD-Net decreases with the increase in the sampling step number. This can be attributed to the excessive number of denoising steps, which may cause CSD-Net to generate more accurate but less diverse descriptions.

[0093] Table 3

[0094]

[0095] Table 3. Diversity evaluation of the top 5 sentences following the consistency reranking procedure on the m-RNN test split of MSCOCO, using the diversity metric.

Claims

1. A diversified image description generation method based on a cascaded self-conditional diffusion network, characterized in that: It includes several cascaded sub-conditional diffusion modules. Each level of sub-conditional diffusion module obtains the prediction noise of the sub-conditional diffusion module of this level by combining the visual tag, word vector, and the prediction noise of the previous level of sub-conditional diffusion module according to the training target. The prediction noise of the last level of sub-conditional diffusion module obtains the final output noise according to the trainable language model head. ,according to Gets the image description.

2. The method according to claim 1, characterized in that Each level of sub-conditional diffusion module includes the following steps: Step S100, obtaining Q visual markers of the image feature V, compressing the two-dimensional matrix of the visual markers through a two-dimensional adaptive average pooling operation, transforming the compressed two-dimensional matrix into a one-dimensional array and aligning it with the image embedding to obtain V'; Step S200, obtaining the predicted noise of the i-th conditional diffusion module according to the training target .

3. The method according to claim 2, characterized in that The specific process of step S100 includes: Step S101, obtaining Q visual labels of image features V; Step S102, rearrange the Q visual markers into a dimension of A two-dimensional matrix of Step S103, using a two-dimensional adaptive average pooling operation (2D AvgPool) to transform the dimension The two-dimensional matrix is ​​compressed to the dimension A collection of compressed tags; Step S104, converting the compressed set of compressed markers into a one-dimensional array containing P markers; Step S105, embedding the one-dimensional array and the image through linear transformation alignment, as a query for the tag projector, d I is the dimension of image embedding; Step S106, the mark projector maps the aligned data to the same space as the aligned image embedding to obtain V'.

4. The method according to claim 3, characterized in that The specific process of step S200 includes: Step S201: According to the forward process of formula (3), at any time step t, and Sampling Noise Data （3） Among them, d i is the i-th sub-conditional diffusion module, is the word vector, Reconstruct the result for the i-1th sub-conditional diffusion module, represents the Gaussian noise associated with time step t, and I is the identity matrix; Step S202, embed the time into t emb With noisy data Add element by element; Step S203, for time embedding and V', according to the posterior distribution prediction of formula (4) （4） in in, .

5. The method according to claim 4, characterized in that Use reparameterization to train the sampled word vectors on the prior conditional distribution of formula (5) (5) Step by step to word vector Add Gaussian noise (6) Until the diffusion step T, the sampled Approximately Gaussian distribution.

6. The method according to claim 5, characterized in that By closed form with random time steps t and sampling , formula (6) is adjusted to （7）。

Images