Image Diversification Description Method Based on Conditional Variational Autoencoder and Contrastive Learning

Abstract

The present invention provides an image diversified description method based on conditional variational autoencoder and contrastive learning, comprising: inputting each image-description pair in a training set into a single-branch conditional variational autoencoder and decoder network model for pre-training to obtain a pre-trained sequence latent variable; introducing another single-branch conditional variational autoencoder and decoder network, calculating a variational lower bound loss function, and obtaining an encoded sequence latent variable through the branch conditional variational autoencoder; introducing a word-level bidirectional contrastive learning loss function to improve the sequence latent variable z c word-level discrimination ability; obtain a global contrast learning objective function; obtain a joint optimization goal to improve the latent representation discrimination ability and fine-grained description sentence decoding ability of the image diversity model, thereby obtaining high-quality diversified description sentences.

Description

Technical Field

[0001] The invention relates to a digital data processing technology, in particular to an image diversified description method based on conditional variational autoencoding and contrastive learning. Background Art

[0002] The image description task is a basic multimodal task at the intersection of computer vision and natural language processing. Early image description models ignored the diversity of generated descriptions and only focused on accuracy, resulting in the generated descriptions being simple and repetitive, which does not conform to the richness of human language. Therefore, diversified image description has become a research hotspot that more and more researchers are paying attention to. The goal of this task is to generate multiple description sentences with diverse words and sentences for a given image while ensuring accurate descriptions. Patent application number 202211628528.4 provides an interpretable text classification system based on a dual-path encoder. The patent discloses a semantic representation that fuses multi-head attention with bidirectional gated recurrent units to solve the problem of mismatch between query and attention results; however, each latent variable in the patent is directly generated by a different single branch, so the latent variables generated by these models lack the distinction between paired and unpaired image-descriptions, resulting in the descriptions generated by different images being similar. Summary of the invention

[0003] The object of the present invention is to provide a method for diversified image description based on conditional variational autoencoding and contrastive learning, comprising:

[0004] Step S100, input each image-description pair in the training set into a single-branch conditional variational autoencoder and decoder network model for pre-training to obtain the pre-trained sequence latent variable z b ;

[0005] Step S200, introduce another single-branch conditional variational autoencoder and decoder network, calculate the variational lower bound loss function L1(x,I), and obtain the encoded sequence hidden variable z through the branch conditional variational autoencoder c ;

[0006] Step S300, introducing word-level bidirectional contrastive learning loss function L r (z b ,z c ), improve the sequence hidden variable z c Word-level discrimination ability;

[0007] Step S400, using the pre-trained sequence latent variable z b Generate the corresponding description statement C N , using the sequence hidden variable z cGenerate the corresponding description statement C G , using the real statement C T And the cross entropy loss function for C G Perform sentence-level global contrastive learning to obtain the global contrastive learning objective function L2(x,I);

[0008] Step S500, obtaining the joint optimization target L total =λ1L1(x,I)+λ2L2(x,I)+λ3L r (z b ,z c ), improve the implicit representation discrimination ability and fine-grained description sentence decoding ability of the image diversity model, so as to obtain high-quality diversified description sentences, where λ1, λ2 and λ3 are weight parameters for balancing the loss of each part.

[0009] Furthermore, in step S200, the variational lower bound loss function L1(x,I) is obtained by the following formula:

[0010]

[0011] in, represents the log-conditional likelihood of the decoder generating the sentence expectations, Represents the prior model at all time steps and the posterior model p θ (z t z ＜t ,x ＜t ,I), θ is the model parameter, x represents the description of T length, x t is the word generated in step t, z is the hidden variable, z t is the hidden variable of the tth step, t∈T, and I represents the image.

[0012] Furthermore, in step S300, the word-level bidirectional contrastive learning loss function L r (z b ,z c ) is obtained by the following formula

[0013]

[0014] Where m represents the positive boundary, (z b ,z c ) denotes the paired latent variables obtained from encoding the same image-description pairs in the same batch by the dual-branch conditional variational autoencoder, (z b ',z c ) and (z b ,z c ') are unpaired latent variables in the same batch.

[0015] Further, in step S400, the global contrastive learning objective function L2(x,I) is obtained by the following formula:

[0016]

[0017] Among them, L XE represents the cross entropy loss, K represents the batch size, and α is a hyperparameter.

[0018] Compared with the prior art, the present invention has the following advantages: the dual-branch conditional variational autoencoder provided by the present invention significantly improves diversity while ensuring the accuracy of model generation description by combining sequence variational autoencoder with contrastive learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the method of the present invention.

[0020] Figure 2 Schematic diagram of a graphical model for word generation in the sequence latent space of the present invention.

[0021] Figure 3 Illustration of a qualitative comparison of descriptions generated for different image captioning models.

[0022] Figure 4 A qualitative example of DS-CVAE generating descriptions based on the test set. DETAILED DESCRIPTION

[0023] Combination Figure 1 , a method for diversified image description based on conditional variational autoencoder and contrastive learning, comprising the following steps:

[0024] Step S100, input each image-description pair in the training set into a single-branch conditional variational autoencoder and decoder network model for pre-training to obtain the pre-trained sequence latent variable z b ;

[0025] Step S200, introduce another single-branch conditional variational autoencoder and decoder network, calculate the variational lower bound loss function L1(x,I), and obtain the encoded sequence hidden variable z through the branch conditional variational autoencoder c ;

[0026] Step S300: Input the image-description pairs in the same training batch into the conditional variational autoencoders corresponding to the two branches respectively to obtain the sequence latent variable z b and z c , introduce word-level bidirectional contrastive learning loss function L r (z b ,z c), improve the sequence hidden variable z c Word-level discrimination ability;

[0027] Step S400, using the pre-trained sequence latent variable z b Generate the corresponding description statement C N , using the sequence hidden variable z c Generate the corresponding description statement C G , using the real statement C T And the cross entropy loss function for C G Perform sentence-level global contrastive learning to obtain the global contrastive learning objective function L2(x,I);

[0028] Step S500, obtaining the joint optimization target L total =λ1L1(x,I)+λ2L2(x,I)+λ3L r (z b ,z c ), where λ1, λ2 and λ3 are weight parameters for balancing the loss of each part. Through word-level contrast loss, sentence-level contrast loss and variational lower bound loss function, the latent representation discrimination ability and fine-grained description sentence decoding ability of the image diversification model are improved, thereby obtaining high-quality diversified description sentences.

[0029] Specifically, the purpose of the present invention is to generate a variety of different description sets X for a given image I. By maximizing the conditional probability p θ (x,zI) achieves this goal.

[0030]

[0031] Among them, θ is the model parameter, x represents the description of T length, and x t is the word generated in step t, z is the hidden variable, z t is the hidden variable of the tth step, t∈T.

[0032] In practice, the long short-term neural network LSTM is used to establish the conditional probability distribution p θ (xI)

[0033]

[0034] The hidden layer state h t-1 contains all x before the current time step ＜t information, and x t-1 Will directly affect the current time step x t However, using the word sequence generated before the current time step and the conditional probability distribution p θ (x t x ＜t,I) can only achieve a one-to-one mapping between images and descriptions, and cannot generate diversified description sentences. To this end, this embodiment introduces a hidden variable z t , the latent variable provides the model with more diverse choices of generated words at each time step. At the same time, the variational lower bound logp required by the conditional variational autoencoder θ (xI)

[0035]

[0036] In formula (5), the first term represents the log-conditional likelihood of the decoder generating the sentence The second term represents the expectation of the prior model at all time steps and the posterior model p θ (z t z ＜t ,x ＜t ,I). The prior model and the posterior model correspond to the encoding network of the dual-branch conditional variational autoencoder.

[0037] In step S100, the conditional variational autoencoder and decoder network are trained by maximizing the variational lower bound of the conditional probability distribution, that is, the lower bound loss function L1(x,I), so that the prior network approaches the posterior network and outputs the pre-trained latent variable z b Given an image I, the prior model can be parameterized as the product of a series of conditions p θ (z t z ＜t ,x ＜t ,I). In the test phase, this embodiment proposes a conditional variational autoencoder (DS-CVAE) that samples latent variables z from the prior model t As the condition for the decoder to generate the corresponding word. Specifically, at each time step t, the hidden variable z t The sampling is based on the x of all previous time steps ＜t and z ＜t , then sample the latent variable z ＜t Same as all x ＜t Jointly predict the word x at the current time step ＜t Therefore, DS-CVAE can be obtained from p θ (z t z ＜t ,x ＜t ,I) in which a series of diverse latent variables z are sampled t And input to the decoder p θ (x t x ＜t ,z ＜t ,I) realizes the diversity of generated descriptions, such as Figure 2 shown.

[0038]

[0039] Furthermore, the pre-trained latent variable z b There is a lack of distinction between paired and unpaired image-description pairs. Therefore, in this embodiment, for each image-description pair in the training set, it is input into a dual-branch conditional variational autoencoder for encoding, and the diversity and distinguishability of latent variables are improved through comparative learning between the two branches. On this basis, step S300 uses formula (7) for joint training

[0040]

[0041] Where m represents the positive boundary; (z b ,z c ) denotes the paired latent variables obtained from encoding the same image-description pairs in the same batch by the dual-branch conditional variational autoencoder, (z b ',z c ) and (z b ,z c ') is an unpaired latent variable in the same batch. The first max term in formula (7), z c Sampling from Figure 2 The posterior model of the middle and lower branch Seq-CVAE, z b and z b ' is the posterior model sampled from the pre-trained upper branch Seq-CVAE; z b Indicates that z c are positive samples sampled from the same paired image-description pair, and z b ' represents a random sampling of unpaired image-description pairs from the current batch. The latter max term in (7) is the opposite of this.

[0042] The contrastive learning in step S300 is essentially a local contrastive learning, which aims to improve the distinguishability of the latent variables corresponding to each word in the latent space.

[0043] In order to further eliminate the repetitive words and phrases in the image description and alleviate the deviation in the training and testing stages, this embodiment adopts the cross entropy loss method to perform global contrastive learning. The specific process of step S400 is as follows:

[0044] Step S401, the description predicted by the decoder is regarded as a positive sample, and the description sampled by the pre-trained decoder is defined as a negative sample;

[0045] Step S402, using positive and negative sample pairs, combining global contrastive learning and cross entropy loss to design a global contrastive loss L2(x,I) for global contrastive learning;

[0046]

[0047] Among them, L XE represents the cross entropy loss; K represents the batch size; α is a hyperparameter; C T and C N Represents the true sentence of each given image I and the negative sample sentence generated by the pre-training model; C G It is a description sentence generated by the current model decoder through a greedy strategy. The greedy strategy provides the generated words to the decoder one by one, and predicts the next word with the largest probability value in turn according to the cross entropy loss; from the sentence level, L2(x,I) uses a mixed training objective to reduce the decoder's generation of common words, and at the same time encourages the decoder to generate accurate and diverse words contained in the positive samples.

[0048] Embodiment 1

[0049] DS-CVAE uses Faster-RCNN to extract 2048-D target features for each image. Then the Seq-CVAE encoding network and decoder are pre-trained for 1630 iterations. In the joint training stage, the dual-branch Seq-CVAE is trained for 20375 iterations. Among them, the dimension of the latent variable z is set to 128. The decoder takes the concatenated vector of image features and sequence latent variables as input at each time step. DS-CVAE uses the SGD optimizer, where the learning rate is set to 0.015, the momentum is set to 0.9, and the weight decay is set to 0.001. To further highlight the role of global contrastive learning, m in Equation 6 is set to 0.2, and α in Equation 7 is set to 1.5. λ1, λ2, and λ3 in the joint optimization objective function are set to 1, 1, and 1.2, respectively.

[0050] (1) Best-1 Accuracy

[0051] Table 1 shows the performance comparison of DS-CVAE on the MSCOCO dataset "M-RNN" test set using Oracle re-ranking accuracy (comparison between sampling 20 and sampling 100). DS-CVAE significantly outperforms the current image diversification description method at sampling 100, and is comparable to the current best performing baseline model COS-CVAE at sampling 20. In terms of SPICE indicators, DS-CVAE achieves the best results compared to other models, such as scores of 0.294 and 0.337 at sampling 20 and 100, respectively, which shows that DS-CVAE prefers to generate fine-grained and discriminative words or phrases rather than common words or phrases.

[0052] Table 1

[0053]

[0054]

[0055] (2) Diversity evaluation

[0056] In order to evaluate the DS-CVAE model more comprehensively, this embodiment compares the diversity indicators of the DS-CVAE model as shown in Table 2. For each diversity evaluation indicator, the best five sentences are selected based on the re-ranking of Consensus.

[0057] As can be seen from Table 2, DS-CVAE achieves good diversity evaluation results. DS-CVAE outperforms AG-CVAE and POS methods when sampling 20 and 100 samples. When sampling 20 samples, DS-CVAE outperforms the baseline models Seq-CVAE and COS-CVAE in all indicators except mBleu. When sampling 100 samples, DS-CVAE achieves the best results in three of the five diversity indicators. In particular, for the Div-1 and Div-2 indicators, DS-CVAE performs significantly better than all other models. When sampling 20 and 100 samples, DS-CVAE improves the Div-2 score by 25% and 10% respectively compared to the COS-CVAE model. The Div-1 and Div-2 scores show that DS-CVAE is able to generate more fine-grained and diverse words and word combinations.

[0058] Table 2

[0059]

[0060] (3) Ablation experiment analysis

[0061] This example analyzes the effectiveness of each component of the DS-CVAE model. Specifically, the bidirectional contrastive learning in the encoding phase and the global contrastive learning in the decoding phase are experimentally verified. Tables 3 and 4 are the ablation experiment results of accuracy and diversity when sampling 20 and 100 samples, respectively, where the baseline model is the pre-trained Seq-CVAE model without the introduction of contrastive learning.

[0062] Table 3

[0063]

[0064] Table 4

[0065]

[0066] As can be seen from the table, compared with the baseline model, the two-way contrastive learning used in the encoding stage has significantly improved both accuracy and diversity indicators. Therefore, it can be proved that two-way contrastive learning is crucial to improving the diversity and distinguishability of latent variables. In the decoding stage, the global contrastive learning has a slight decrease in accuracy when sampling 20 and 100 samples, but the diversity is significantly increased. Therefore, DS-CVAE with global contrastive learning can achieve a balance between accuracy and diversity.

[0067] (4) Qualitative analysis of model performance

[0068] In order to more comprehensively evaluate the diversity and accuracy of the DS-CVAE model, this example compares it with the descriptions generated by existing image diversified description methods.

[0069] like Figure 3 As shown in the figure, for the first image, COS-CVAE generates common n-grams, such as "on the top of", but does not reflect the details in the image description, such as the color of the cat. Although AG-CVAE describes some details of the image, such as "small gray and white", it still cannot avoid generating common words or phrases. Div-BS, SeqCVAE, and POS also tend to generate accurate and common words and phrases, and cannot strike a balance between diversity and accuracy.

[0070] The descriptions generated by the DS-CVAE method proposed in this example are not only grammatically correct but also contain more details of the image, such as "whitecat" and "redsuitcase". In particular, DS-CVAE generates verbs that appear less frequently in the dataset but are more meaningful, such as "resting" and "looking", rather than more frequent verbs, such as "sit" and "laying". Figure 3For the second image in the image, the description generated by Seq-CVAE contains some repeated phrases, such as "somewaternearwater", and lacks description of the image details. The descriptions of the image by POS and Div-BS are very similar. The DS-CVAE model proposed in this embodiment not only describes the image with correct grammar, but also generates more diverse and accurate descriptions. Specifically, DS-CVAE even generates complex sentences, such as "that" clauses, which are obviously more in line with human language habits. In addition, the descriptions generated by DS-CVAE usually contain important and detailed descriptions of the image. For example, the DS-CVAE model describes "redbeak" that does not appear in any baseline model. With its flexible grammar and word usage, DS-CVAE outperforms the baseline models in terms of balancing diversity and accuracy.

[0071] exist Figure 4 In this embodiment, some examples generated by DS-CVAE for various test set images are further shown. As can be seen from the figure, for a given image, most of the descriptions generated by DS-CVAE are natural, fluent and diverse. DS-CVAE is able to describe the specific details of a given image, such as "bluejacket", "holdingapairofskis", etc. For images of airplanes, DS-CVAE gives corresponding diverse and detailed descriptions, for example, "blackairplane", "militaryairplane", "smallfighter jet" and "blackfighter". Different descriptions of the same object are also more in line with the diversity characteristics of human natural language. In addition, compared with Figure 3 Similar to the examples in , DS-CVAE generates more meaningful and flexible verbs, such as "holding", "wearing", "permit", and "arrange", which makes the description more distinctive. Therefore, based on quantitative and qualitative experimental analysis, the DS-CVAE proposed in this embodiment outperforms the existing image diversification description methods in terms of diversity and accuracy.

Claims

1. A method for diversified image description based on conditional variational autoencoder and contrastive learning, characterized in that: include: Step S100, input each image-description pair in the training set into a single-branch conditional variational autoencoder and decoder network model for pre-training to obtain the pre-trained sequence latent variable z b ; Step S200, introduce another single-branch conditional variational autoencoder and decoder network, calculate the variational lower bound loss function L1(x,I), and obtain the encoded sequence hidden variable z through the branch conditional variational autoencoder c ; Step S300, introducing word-level bidirectional contrastive learning loss function L r (z b ,z c ), improve the sequence hidden variable z c Word-level discrimination ability; Step S400, using the pre-trained sequence latent variable z b Generate the corresponding description statement C N , using the sequence hidden variable z c Generate the corresponding description statement C G , using the real statement C T And the cross entropy loss function for C G Perform sentence-level global contrastive learning to obtain the global contrastive learning objective function L2(x,I); Step S500, obtaining the joint optimization target L total =λ1L1(x,I)+λ2L2(x,I)+λ3L r (z b ,z c ), improve the latent representation discrimination ability of the image diversity model and the fine-grained description sentence decoding ability, so as to obtain high-quality diversified description sentences, where λ1, λ2 and λ3 are weight parameters for balancing the loss of each part; In step S300, the word-level bidirectional contrastive learning loss function L r (z b ,z c ) is obtained by the following formula Where m represents the positive boundary, (z b ,z c ) denotes the paired latent variables obtained from encoding the same image-description pairs in the same batch by the dual-branch conditional variational autoencoder, (z b ',z c ) and (z b ,z c ') are unpaired latent variables in the same batch; In step S200, the variational lower bound loss function L1(x,I) is obtained by the following formula: in, represents the log-conditional likelihood of the decoder generating the sentence expectations, Represents the prior model at all time steps and the posterior model p θ (z t z ＜t ,x ＜t ,I), θ is the model parameter, x represents the description of T length, x t is the word generated in step t, z is the hidden variable, z t is the hidden variable of the tth step, t∈T, I represents the image; In step S400, the global contrastive learning objective function L2(x,I) is obtained by the following formula: Among them, L XE represents the cross entropy loss, K represents the batch size, and α is a hyperparameter. and They represent the true sentence of the i-th given image I and the negative sample sentence generated by the pre-trained model, respectively. It is the description sentence of the i-th given image I generated by the current model decoder through the greedy strategy.

Images