A diversified image description method based on conditional variational transformer and introspective adversarial learning

Abstract

This invention discloses a diversified image description method based on Conditional Variational Transformer and Introspective Adversarial Learning, comprising: extracting regional features of an image to obtain an input vector, which is then input into a pre-defined Conditional Variational Transformer model to obtain corresponding descriptive statements for the image; the model includes a variational inference encoder and a generator; pre-training the model using the variational lower bound mechanism in sequential conditional autoencoders; and then fine-tuning it using an introspective adversarial learning strategy. Without adding an additional discriminator, the generated descriptive statements are self-evaluated through adversarial learning to achieve diversified image descriptions. This invention employs a sequential conditional variational autoencoder architecture combined with a Conditional Variational Transformer model, solving the problem of the model being limited by a single mapping; and uses an introspective adversarial learning strategy to alleviate the problem of the model being limited by human evaluation metrics, thereby improving the performance of diversified image descriptions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and natural language processing, and particularly relates to a diversified image description method based on conditional variational Transformer and introspective adversarial learning. BACKGROUND

[0002] Image description generation, as a basic research topic of multi-modal machine learning, aims to enable machines to understand the content of images and automatically generate texts corresponding to images in the form of natural language. It is very easy for humans to describe the content of a picture, but it is extremely challenging for machines. Meanwhile, image description has important scientific research and application value. In the field of image retrieval and classification, it helps to improve the accuracy of image content retrieval and classification. In the field of early childhood education, it helps children grow quickly and perceive the world when combined with speech recognition technology. In the field of visual assistance, it helps visually impaired people understand image information and make up for visual defects. In the field of intelligent medical treatment, it can automatically generate medical image diagnosis reports and provide technical support for intelligent diagnosis and treatment.

[0003] With the significant breakthroughs of deep learning in the field of artificial intelligence, image description based on deep encoder-decoder network has become the mainstream method at present. This kind of method usually uses convolutional neural network (CNN) as an image encoder to extract visual features, and then uses recurrent neural network (RNN) as a decoder to convert visual features into corresponding description texts.

[0004] In recent years, the Transformer architecture in natural language processing has begun to be introduced into the field of image description, and has achieved better image description performance than CNN-RNN models. Although the Transformer model improves the indicators of image description generation to some extent, it is limited by the "black box" mapping of deep learning, and the model is limited by the problems of single mapping and artificial evaluation indicators, and tends to generate unique descriptions composed of simple repetitive phrases, which cannot guarantee the diversity of the description. SUMMARY

[0005] The technical problems solved: In view of the above technical problems, the application provides a diversified image description method based on conditional variational Transformer and self-reflection adversarial learning, adopts a serialized conditional variational auto-encoding architecture, and is combined with a Transformer-based image description method, thereby solving the problem that the model is limited to a single mapping; the self-reflection adversarial learning strategy is adopted, the generated description sentences are self-evaluated under the premise of not adding a discriminator, thereby alleviating the problem that the model is limited to artificial evaluation indicators, so as to improve the performance of diversified image description.

[0006] Technical scheme: A diversified image description method based on conditional variational Transformer and self-reflection adversarial learning comprises the following steps:

[0007] Step 1, obtaining an image and extracting the region features of the image to obtain an input vector;

[0008] Step 2, inputting the above vector into a preset conditional variational Transformer model to obtain the corresponding description sentences of the image; wherein the preset conditional variational Transformer model comprises a variational inference encoder and a generator, and the variational inference encoder and the generator are both composed of a mask self-attention layer, a cross-attention layer, a normalization residual layer and a feedforward neural network;

[0009] Step 3, pre-training the above preset conditional variational Transformer model using the variational lower bound mechanism in the serialized conditional auto-encoding;

[0010] Step 4, fine-tuning the above pre-trained model using a self-reflection adversarial learning strategy, under the premise of not adding a discriminator, self-evaluating the generated description sentences through the adversarial learning idea, to realize the diversification of image description.

[0011] Preferably, the image in step 1 is extracted using a Faster R-CNN target detection algorithm.

[0012] Preferably, the vector in step 1 is obtained by reducing the dimension of the region features through a linear layer.

[0013] Preferably, the specific steps of step 2 are as follows:

[0014] Step 2-1, inputting the vector into the variational inference encoder, and first passing through a feature coding network in the variational inference encoder to obtain visual and text mixed semantic features, and then inputting the visual and text mixed semantic features into a posterior inference subnetwork and a prior approximation subnetwork to obtain an encoding result;

[0015] Step 2-2, input the encoding result into the generator to obtain a description sentence corresponding to the image.

[0016] Further, in step 2-1, the feature encoding network is composed of N layers of attention modules.

[0017] Further, in step 2-1, the visual and text mixed semantic features are input into the posterior inference subnetwork for calculation to obtain a first calculation result W q ; wherein the calculation formula is:

[0018] W q =LayerNorm(MultiHead(W 0 , W 0 , W 0 )+W 0 )

[0019] wherein, represents an input vector obtained by performing position coding on a word embedding vector;

[0020] The first calculation result and the attention result are interacted through a cross-attention layer and a normalization residual layer to obtain a fused feature F q ; wherein the calculation formula is:

[0021] F q =LayerNorm(MultiHead(W q , V N , V N )+W q )

[0022] wherein, M is the number of region features of the image, and T is the number of words;

[0023] The feature F q is input into the feedforward neural network in the prior approximation subnetwork and the posterior inference subnetwork to generate the mean and variance, and the posterior latent variable is sampled through the reparameterization technique.

[0024] Further, in step 2-1, the attention result is input into the prior approximation subnetwork for calculation to obtain a second calculation result W p ; wherein the calculation formula is:

[0025] W p =LayerNorm(MaskedMultiHead(W 0 , W 0 , W 0 )+W 0 ) wherein,

[0026] The second calculation result and the attention result are interacted through a cross attention layer and a normalized residual layer to obtain a fused feature F p ; wherein the calculation formula is:

[0027] F p = LayerNorm (MultiHead (W p , V N , V N + W p )

[0028] wherein, M is the number of region features of the image;

[0029] The feature F p is respectively input into feedforward neural networks in the prior approximation subnetwork and the posterior inference subnetwork to generate mean and variance, and prior latent variables are sampled through a reparameterization technique.

[0030] Furthermore, in the model training stage, the encoding result is: the posterior latent variable is spliced with the feature F p after dimension reduction by the full connection layer; in the model testing stage, the encoding result is: the prior latent variable is spliced with the feature F p after dimension reduction by the full connection layer.

[0031] Preferably, in the step 3, the pre-training process is: the cross-entropy loss is used to measure the similarity between the generated description and the artificial annotation description, and the KL loss is used to regularize the variational inference encoder by minimizing the KL divergence between the approximate posterior and the conditional prior.

[0032] Preferably, in the step 4, the fine-tuning of the self-reflection adversarial learning strategy adopts the following formula:

[0033]

[0034]

[0035] wherein [·] + = max (0, ·), m is a positive margin, sg (·) indicates that back propagation is stopped at this time, and α and β are weighting parameters, L XE represents the cross-entropy loss, L KL represents the KL loss, x g represents the artificial annotation true description sentence, and x p represents the description sentence sampled by the generator.

[0036] Beneficial effects: the application realizes one-to-many mapping of image to text by combining the serialized conditional encoding architecture with the Transformer model, realizes generation of any number of descriptions by sampling any number of hidden space vectors as a given image condition, and can effectively solve the problem that the current Transformer model in image description is limited to single mapping. At the same time, a new self-reflection adversarial learning strategy applied to the text generation task is designed, mainly aiming at the problems of text discrete difficulty in back propagation and model limitation to artificial evaluation, and the generated description is returned to the encoder for adversarial learning through the gumbe1 softmax skill, so as to further optimize the quality of the generated description sentence. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The architecture diagram of the conditional variational Transformer model in the application;

[0038] Figure 2 The training flowchart of the self-reflection adversarial learning strategy in the application;

[0039] Figure 3 The method flowchart of the embodiment of the application. DETAILED DESCRIPTION

[0040] The application will be further described below in combination with the drawings and specific embodiments.

[0041] Embodiment 1

[0042] As shown in Figure 1 and Figure 3 , a diversified image description method based on conditional variational Transformer and self-reflection adversarial learning includes the following steps:

[0043] Step 1, an image is obtained, and the region features of the image are extracted by using the Faster R-CNN target detection algorithm, and the region features are reduced in dimension through a linear layer to obtain an input vector.

[0044] Step 2, input the above vector into a preset conditional variational Transformer model to obtain the corresponding description sentence of the image; wherein the preset conditional variational Transformer model includes a variational inference encoder and a generator, and the variational inference encoder and the generator are both composed of a mask self-attention layer, a cross-attention layer, a normalization residual layer and a feedforward neural network.

[0045] Step 2-1, input the vector into the variational inference encoder, and first pass through a feature encoding network in the variational inference encoder to obtain visual and text mixed semantic features, then input the visual and text mixed semantic features into a posterior inference subnetwork and a prior approximation subnetwork to obtain an encoding result.

[0046] Specifically, the vector is input into a feature encoding network (which is an encoder composed of N attention modules), and after operations such as attention and residual normalization, a hybrid semantic feature V combining visual and textual elements is obtained. N ;

[0047] The visual and textual hybrid semantic features are input into the posterior inference subnetwork for calculation, resulting in the first calculation result W. q The calculation formula is as follows:

[0048] W q =LayerNorm(MultiHead(W 0 W 0 W 0 )+W 0 )

[0049] in, This represents the input vector obtained by positional encoding of the word embedding vector;

[0050] The first calculation result and the attention result are interacted through a cross-attention layer and a normalized residual layer to obtain the fused feature F. q The calculation formula is as follows:

[0051] F q =LayerNorm(MultiHead(W q V N V N )+W q )

[0052] in, M is the number of region features in the image, and T is the number of words;

[0053] The feature F q The feedforward neural networks in the prior approximation subnetwork and the posterior inference subnetwork are respectively input to generate the mean and variance, and the posterior latent variable z is sampled using a reparameterization technique. t The calculation formula is:

[0054]

[0055] in Two feedforward neural networks are used to generate the mean and variance, respectively, and the reparameter trick z is applied. t =μ t +σ t ⊙∈,∈~N(0,I)from N(z) t μ t , σ t z was obtained by sampling t;

[0056] The visual and text mixed semantic features are input into a prior approximation subnetwork for calculation to obtain a second calculation result W p ; wherein the calculation formula is:

[0057] W p =LayerNom(MaskedMultiHead(W 0 , W 0 , W 0 )+W 0 )

[0058] wherein,

[0059] The second calculation result and the attention result are interacted through a cross-attention layer and a normalization residual layer to obtain a fused feature F p ; wherein the calculation formula is:

[0060] F p =LayerNorm(MultiHead(W p , V N , V N )+W p )

[0061] wherein, M is the number of region features of the image;

[0062] The feature F p is input into a feedforward neural network in the prior approximation subnetwork and the posterior inference subnetwork respectively to generate a mean and a variance, and a prior latent variable z t is sampled through a reparameterization technique; the calculation formula is:

[0063]

[0064] wherein the prior latent variable z t is a multivariate Gaussian distribution conforming to a mean μ t (z t-1 , x ＜t , I) and a standard deviation σ t (z t-1 , x ＜t , I);

[0065] Step 2-2, input the encoding result into the generator to obtain a description sentence corresponding to the image; wherein the generator is composed of a standard Transformer decoder and a Gumbel sampler. In the model training stage, the encoding result is: the feature F p that is fused by the posterior latent variable, the visual feature and the text feature in the prior approximation subnetwork.The concatenated operation and full connection layer dimension reduction are performed on the 512-dimensional features; in the model test stage, the encoding result is that the prior hidden variable is combined with the feature F p The concatenated operation and full connection layer dimension reduction are performed on the 512-dimensional features.

[0066] Step 3, the preset conditional variational Transformer model is pre-trained by using the variational lower bound mechanism in the serialized conditional self-encoding, an image and its corresponding description text are mapped into a set of hidden variables, and a one-to-many relationship between the image and the description is modeled; the formula is:

[0067]

[0068]

[0069] The cross-entropy loss L XE is used to measure the similarity between the generated description and the artificially annotated description, and the KL loss L KL is used to regularize the variational inference encoder by minimizing the KL divergence between the approximate posterior and the conditional prior.

[0070] Step 4, the pre-trained model is fine-tuned by using the introspective adversarial learning strategy, without adding an additional discriminator, the generated description sentence is self-evaluated by using the adversarial learning idea to realize the diversification of the image description; the formula is as follows:

[0071]

[0072]

[0073] where [·] + = max(0, ·), m is a positive margin, sg(·) represents that back propagation is stopped at this time, and alpha and beta are weighting parameters.

[0074] As shown in Figure 2 , E q and E p are posterior inference subnetworks and prior approximation subnetworks, respectively, for generating posterior and prior hidden variables. The hidden variables are transmitted to the generator to sample description x g and x p . The encoder and the generator are jointly trained by minimizing L KL (I, x) and loss L XE (x, x g ). In addition, when L KL = (I, sg(x g )) ≤ m, the encoder E and the generator G are trained by L E and LG Losses run a minimax game. The encoder simultaneously encodes the image and the description by maximizing L KL (I, sg(x g )), separating the artificially annotated descriptions from the descriptions x g , x p sampled by the generator, while the generator is adversarially trained to generate description sentences with smaller L KL (I, x r ).

[0075] The following simulation experiments are based on the above conditional variational Transformer-based diversified image description method. The implementation method of the present embodiment is described in steps 1-4 above, and the specific steps will not be described in detail. The following only shows the effect of the experimental results.

[0076] In the method, the dimensions of the image features, word embeddings, and latent variables of the conditional variational Transformer model are all 512. Specifically, a pre-trained Faster R-CNN r is used to extract the grid features of each image, and the dimension is 2048 and is linearly mapped to a 512-dimensional vector. In the generator, the word embedding is used and added to the position encoding as the input of the posterior inference subnetwork and the prior approximation subnetwork. In addition, the visual encoder and the generator are both composed of N=3 attention blocks, where the number of heads of the multi-head attention is 8. In the training stage, the experiment uses the Adam optimization algorithm and the warmup learning rate preheating technique to optimize the model, and trains for 30 rounds under the cross-entropy loss. The hyperparameter m is set to 100 according to experience, and the hyperparameters a= b=1.0. In the test stage, the beam width is set to 2 when the accuracy evaluation metric is performed, and the beam width is set to 1 when the diversity evaluation metric is performed. In addition, the generator in the method can use Softmax and Gumbel Softmax operations respectively combined with beam search to sample a group of descriptions.

[0077] Table 1 Performance comparison of accuracy using Oracle reordering on the MSCOCO dataset "M-RNN" test set

[0078]

[0079] Table 2 Performance comparison of diversity using Consensus reordering on the MSCOCO dataset "M-RNN" test set

[0080]

[0081]

[0082] Table 1 compares the performance of ours and other methods in terms of accuracy on MSCOCO "M-RNN" test set after Oracle re-ranking, where "sample" denotes the number of descriptions sampled per image, and the best results are bolded. Oracle re-ranking ranks different descriptions using the maximum score of each metric and selects the best one. Each Oracle evaluation score is the average of the test set. In this experiment, we sample 20 and 100 latent variables and pass them to the generator to decode the description sentences. As shown in Table 1, ours outperforms the compared methods in terms of all accuracy metrics on both 20 and 100 samples.

[0083] Table 2 further evaluates the diversity of the best five sentences of different models on MSCOCO "M-RNN" test set after Consensus re-ranking. As shown in Table 2, ours performs better in all diversity metrics. In particular, ours generates 19.82 / 20 (99.1%) and 97.84 / 1000 (97.8%) unique descriptions. According to the results of 20 and 100 samples, ours is comparable to the state-of-the-art models. Compared with the existing methods, ours has a lower mBLEU-4 score, which indicates that it generates significantly different descriptions. In terms of Div-1 and Div-2, ours significantly outperforms other methods, e.g., it achieves 0.73 and 0.61 in Div-2 on 20 and 100 samples, respectively, which indicates the effectiveness of ours in improving the diversity of image descriptions.

[0084] In summary, ours significantly improves the performance of existing methods in both accuracy and diversity metrics, which sufficiently demonstrates that ours can generate both accurate and diverse descriptions, further indicating the effectiveness of ours.

[0085] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited thereto, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application, which are all within the scope of the claims of the present application.

Claims

1. A diversified image description method based on conditional variational Transformer and self-aware adversarial learning, characterized in that, The method comprises the following steps: Step 1, obtaining an image and extracting regional features of the image to obtain an input vector; Step 2, inputting the vector into a preset conditional variational Transformer model to obtain a corresponding description sentence of the image; wherein the preset conditional variational Transformer model comprises a variational inference encoder and a generator, and the variational inference encoder and the generator are both composed of a mask self-attention layer, a cross-attention layer, a normalization residual layer and a feedforward neural network; Step 3, pre-training the model using the variational lower bound mechanism in the serialized conditional auto-encoding of the above preset conditional variational Transformer model; Step 4, fine-tuning the pre-trained model using introspective adversarial learning strategy, without adding a discriminator, to realize the diversification of image description by self-evaluating the generated description sentence through the adversarial learning idea; wherein the fine-tuning of the introspective adversarial learning strategy uses the following formula: where [·] + = max(0, ·), m is a positive margin, sg(·) indicates backpropagation stops at this point, a and b are trade-off parameters, L XE represents cross-entropy loss, L KL represents KL loss, x g represents a human-labeled true description sentence, x p represents a description sentence sampled by the generator.

2. The method of claim 1, wherein, The image in step 1 is extracted using the Faster R-CNN target detection algorithm.

3. The method of claim 1, wherein, The vector in step 1 is obtained by reducing the dimension of the regional features through a linear layer.

4. The method of claim 1, wherein, The specific steps of step 2 are as follows: Step 2-1, inputting the vector into the variational inference encoder, and first passing through a feature encoding network to obtain visual and text mixed semantic features, and then inputting the visual and text mixed semantic features into a posterior inference subnetwork and a prior approximation subnetwork to obtain an encoding result; Step 2-2, inputting the encoding result into the generator to obtain a description sentence corresponding to the image.

5. The method of claim 4, wherein, In step 2-1, the feature encoding network is composed of N layers of attention modules.

6. The method of claim 4, wherein, In step 2-1, the visual and text mixed semantic features are input into a posterior inference subnetwork for calculation to obtain a first calculation result W q ; wherein the calculation formula is: W q = LayerNorm(MultiHead(W 0 , W 0 , W 0 ) + W 0 ) wherein, represents an input vector obtained after position encoding of the word embedding vector; The first calculation result and the attention result are interacted through a cross-attention layer and a normalized residual layer to obtain a fused feature F q ; wherein the calculation formula is: F q = LayerNorm(MultiHead(W q , V N , V N + W q ) wherein, M is the number of region features of the image, and T is the number of words. The features F q Feedforward neural networks in the prior approximation subnetwork and the posterior inference subnetwork, respectively, generate the mean and variance, and sample the posterior latent variables through the reparameterization trick.

7. The method of claim 6, wherein, In step 2-1, the visual and text mixed semantic features are input into a prior approximation subnetwork for calculation to obtain a second calculation result W p ; wherein the calculation formula is: W p = LayerNorm(MaskedMultiHead(W 0 , W 0 , W 0 ) + W 0 ) wherein The second calculation result and the attention result are interacted through a cross-attention layer and a normalized residual layer to obtain a fused feature F p ; wherein the calculation formula is: F p = LayerNorm(MultiHead(W p , V N , V N + W p ) wherein, M is the number of regional features of the image; The features F p The feedforward neural networks in the prior approximation subnetwork and the posterior inference subnetwork respectively generate the mean and variance, and sample the prior latent variable through the reparameterization trick.

8. The method of claim 7, wherein, In the model training stage, the encoding result is: the posterior latent variable is combined with the feature F p The feature after the concatenation operation and the full connection layer dimension reduction; in the model testing stage, the encoding result is: the prior latent variable is combined with the feature F p The feature after the concatenation operation and the full connection layer dimension reduction.

9. The method of claim 1, wherein, In step 3, the pre-training process is as follows: the cross-entropy loss is used to measure the similarity between the generated description and the manually labeled description, and the KL loss is used to regularize the variational inference encoder by minimizing the KL divergence between the approximate posterior and the conditional prior.

Images