A diverse and controllable image description method based on grammatical structure embedding learning

Abstract

The present invention provides a diverse and controllable image description method based on grammatical structure embedding learning, comprising the following steps: step S100, acquiring visual features of an image and semantic features of a text; step S200, respectively performing a priori approximate subnetwork and a posterior approximate subnetwork learning on the visual features and the text semantic features to obtain prior latent variables and posterior latent variables, and obtaining the KL divergence between the prior latent variables and the posterior latent variables; step S300, using the visual features, the prior latent variables, the posterior latent variables and a series of [MASK] as input, and adopting a non-autoregressive objective function to predict part-of-speech tags; step S400, using the visual feature prior latent variables and the posterior latent variables and the text to generate a description, and obtaining an image description generation optimization target; step S500, optimizing the loss function composed of the non-autoregressive target, the image description generation optimization target and the KL divergence to obtain a neural network that generates an optimal image description.

Description

Technical Field

[0001] The present invention relates to an image data processing technology, and in particular to a diverse and controllable image description method based on grammatical structure embedding learning. Background Art

[0002] Advanced encoder-decoder frameworks and Transformer models have improved the accuracy of image description generation models. However, because accuracy-oriented models (such as models based on reinforcement learning (RL) to optimize the CIDEr metric) tend to learn the most common sentence patterns in the training corpus, the generated image descriptions often contain common words and phrases, which can easily cause model collapse. Currently, solutions to the model collapse problem mainly focus on two approaches: (1) Diversified Image Caption Generation (DIC), that is, improving the diversity of generated descriptions; and (2) Controllable Image Caption Generation (CIC), that is, enhancing the controllability of image descriptions.

[0003] Although DIC models can alleviate model collapse, they lack the ability to control the caption generation process. Existing CIC models use part-of-speech tagging information or discrete sentence patterns as control signals to guide description generation. However, these methods predefine a set of patterns and construct a pattern encoding network for images to obtain control signals. As a result, the flexibility and diversity of the description generation model are limited by the predefined discrete patterns. Summary of the Invention

[0004] The present invention aims to provide a diverse and controllable image description method based on grammatical structure embedding learning, comprising the following steps:

[0005] Step S100, obtaining visual features of image I and semantic features of text;

[0006] Step S200 , performing learning of a priori approximate subnetwork and a posterior approximate subnetwork on the visual features and the text semantic features respectively to obtain a priori latent variable z and a posterior latent variable z~, and obtaining the KL divergence between the priori latent variable z and the posterior latent variable z~;

[0007] Step S300 , using visual features, prior latent variables z and posterior latent variables z~ and a series of [MASK] as input, and using a non-autoregressive objective function to predict part-of-speech tags;

[0008] Step S400, using the visual feature prior latent variable z and the posterior latent variable z~ and the text to generate a description, and obtaining an optimization target for image description generation;

[0009] Step S500 , optimizing the loss function composed of the non-autoregressive objective, the image description generation optimization objective, and the KL divergence, to obtain a neural network for generating the optimal image description.

[0010] Furthermore, the process of learning the prior approximate sub-network in step S200 includes:

[0011] Step S211: Use the multi-head attention module to N The vectors are fused into a single vector U p

[0012] U p =MHA(Q p ,V N ,V N ) (4)

[0013] Among them, V N is the visual feature Q of image I p is a learnable vector;

[0014] Step S212, vector U p Feed it to the feedforward network layer and generate the prior latent variable z according to formula (5)

[0015] p θ (zI)=Ν(z;μ',σ') (5)

[0016] Among them, p θ (zI) is the prior distribution, μ' and σ' are respectively given by FFN(U p ) and FNN is the operation of the feedforward neural network layer.

[0017] Furthermore, the specific process of learning the posterior approximate sub-network in step S200 includes:

[0018] Step S221, using a learnable vector Q q Learn the syntactic information from part-of-speech tags and image features and map them into a global vector U q

[0019] U q =MHA(Q q ,F q ,F q ) (6)

[0020] Among them, Fq is the semantic feature Tq and the visual feature V N Convert to fusion features

[0021] F q =AddNorm(MHA(T q ,V N ,VN ))

[0022] AddNorm represents the residual normalization operation, and MHA represents the operation of the multi-head attention module;

[0023] Step S222, vector U q Feed it to the feedforward network layer and generate the posterior latent variable z~ according to formula (7)

[0024]

[0025] in, and They are FFN(U q ) and FNN is the operation of the feedforward neural network layer.

[0026] Furthermore, the non-autoregressive objective function L1 in step S300 is

[0027]

[0028] Among them, t is the time step, Ψ is the posterior network parameter, s is the part-of-speech tag sequence, s t For each position of the part-of-speech tag, represents the log-likelihood of the data.

[0029] Furthermore, the optimization target L2 in step S400 is

[0030]

[0031] in, represents the log-likelihood of the data, w t The word generated at step t.

[0032] Furthermore, in step S500, the optimal solution of equation (10) is obtained to ensure that L1, L2, L KL Minimum

[0033] L=L1+L2+αL KL (10)

[0034] Among them, α is the trade-off parameter between accuracy and diversity, L KL =D KL (q φ (z|s,I)||p θ (z|I)).

[0035] Furthermore, the process of obtaining the optimal solution to equation (10) in step S500 is as follows:

[0036] Step S501, obtain the conditional distribution p θ (sI)

[0037]

[0038] Among them, s is the part-of-speech tag, z is the latent space;

[0039] Step S502, by maximizing the conditional distribution p θ The log-likelihood of (sI) derives the log-evidence lower bound for part-of-speech tagging

[0040]

[0041] Among them, the first term represents the log-likelihood of the data, which is used to generate the part-of-speech tag sequence, and the second term is the posterior condition and the prior condition p θ KL divergence between (zI);

[0042] Step S503, by maximizing the conditional distribution p Ψ The log-likelihood of (wI) derives the log-evidence lower bound for image description;

[0043] Step S504: Modeling a conditional generative model p Ψ (wI)

[0044]

[0045] The first term on the right side of the inequality sign represents the log-likelihood of the data, and the second term is the posterior parameterized by π. and conditional prior p Ψ The KL divergence between (zI).

[0046] Furthermore, in step S100, the visual feature acquisition method is:

[0047] For the input image I, a set of grid visual features V = {v1, v2, ..., v M}, the grid visual feature V is mapped to V 0 is fed into the standard Transformer encoder and the visual features V of the input image are obtained by formula (1) N

[0048]

[0049] Among them, when l=N-1, the output of the Nth attention block is obtained, that is, V l+1 =V N, MSA represents the multi-head self-attention operation, AddNorm represents the residual normalization operation, FFN represents the feedforward network layer operation, M represents the number of image grids, and d1 represents the dimension of the image feature in the latent space.

[0050] Furthermore, the semantic feature acquisition method is:

[0051] The input text X is added through the embedding layer and position encoding as the input variable T 0 ;

[0052] T 0 is fed into the encoder, which uses a multi-head attention module and residual normalization operation on T 0 Perform self-attention operation to obtain semantic features T q

[0053] T q =AddNorm(MHA(T 0 ,T 0 ,T 0 )) (2)

[0054] in, T represents the length of the text, and d2 represents the dimension of the text feature in the latent space.

[0055] Compared with the prior art, the present invention has the following advantages: the present invention takes into account that the part-of-speech tagging of each word in the description is the basic structure of grammar, uses the part-of-speech tagging sequence to represent the syntactic structure, and learns the global latent space of the sentence patterns of the entire training corpus. It can learn the latent space of syntactic structure embedding under image conditions in an end-to-end manner, generate diverse syntactic structures for each image, and use syntactic structure embedding as a control signal to generate diverse and natural language descriptions.

[0056] The present invention will be further described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0058] Combine Figure 1 , a diverse and controllable image description method based on grammatical structure embedding learning, comprising the following steps:

[0059] Step S100, obtaining visual features of image I and semantic features of text;

[0060] Step S200: The visual feature V and the text semantic feature T are respectively subjected to the prior approximate sub-network and the posterior approximate sub-network learning to obtain the prior latent variable z and the posterior latent variable Get the KL divergence between the prior latent variable z and the posterior latent variable z~

[0061] Step S300 , using visual features, prior latent variables z and posterior latent variables z~ and a series of [MASK] as input, and using a non-autoregressive objective function to predict part-of-speech tags;

[0062] Step S400, using the visual feature prior latent variable z and the posterior latent variable z~ and the text to generate a description, and obtaining an optimization target for image description generation;

[0063] Step S500 , optimizing the loss function composed of the non-autoregressive objective, the image description generation optimization objective, and the KL divergence, to obtain a neural network for generating the optimal image description.

[0064] In step S100, for the input image I, the pre-trained residual neural network ResNet101 (i.e. Figure 1 Image Encoder) to extract a set of grid visual features V = {v1,v2,...,v M}, the grid visual feature V is mapped to V 0 is fed into the standard Transformer encoder, where M represents the number of image grids and d1 represents the dimension of the image features in the latent space.

[0065] Further, if Figure 1 As shown in (a), the standard Transformer encoder includes a multi-head attention module (Multi-Head Attention), two residual normalization modules (Add&Norm) and a feedforward network layer (FeedForward). 0 After being fed into the multi-head attention module, the final visual representation V of the input image is obtained through formula (1): N

[0066]

[0067] Among them, when l=N-1, the output of the Nth attention block is obtained, that is, V l+1 =V N , MSA represents the multi-head self-attention operation, AddNorm represents the residual normalization operation, and FFN represents the feedforward network layer operation.

[0068] In step S100, the process of obtaining semantic features includes: the input text X is added as the input variable T through the embedding layer (Embedding) and positional encoding (Positional Encoding) 0 ;T0 is fed into the encoder.

[0069] Further, such as Figure 1 As shown in (b), the encoder includes two multi-head attention modules (Multi-HeadAttention) and two residual normalization modules (Add&Norm).

[0070] Through formula (2), the first multi-head attention module and residual normalization operation are used to 0 Perform self-attention operation

[0071] T q =AddNorm(MHA(T 0 ,T 0 ,T 0 )) (2)

[0072] in, Among them, T represents the length of the text, and d2 represents the dimension of the text feature in the latent space;

[0073] By using formula (3), T q and V N Convert to fusion feature F q

[0074]

[0075] in,

[0076] In step S200, the specific process of learning the prior approximate sub-network includes:

[0077] Step S211: Use the Multi-Head Attention module to N The vectors are fused into a single vector U p

[0078] U p =MHA(Q p ,V N ,V N ) (4)

[0079] Among them, Q p is a learnable vector;

[0080] Step S212, vector U p Feed it to the feed forward network layer and generate the prior latent variable z according to formula (5)

[0081] p θ(zI)=Ν(z;μ',σ') (5)

[0082] Among them, p θ (zI) is the prior distribution, μ' and σ' are the mean and standard deviation respectively, where μ' and σ' are the values ​​obtained by FFN(U p ) and the mean and standard deviation of z are obtained by the reparameterization technique 0,I) from the distribution Sampling obtained.

[0083] In step S200, the specific process of learning the posterior approximate sub-network includes:

[0084] Step S221, using a learnable vector Q q Learn the syntactic information from part-of-speech tags and image features and map them into a global vector U q

[0085] U q =MHA(Q q ,F q ,F q ) (6)

[0086] Step S222, vector U q Feed it to the feed-forward network layer (Feed Forward), and generate the posterior latent variable z~ according to formula (7)

[0087]

[0088] Among them, s is the part-of-speech tag sequence, and They are FFN(U q ) obtained by the mean and standard deviation, z~ through the re-parameterization technique From the distribution Sampling obtained.

[0089] Combine Figure 1As shown in (c), in step S300, a series of [MASK] are introduced as one of the inputs, and are added to the prior latent variable z and the posterior latent variable z~, and then input into the embedding layer (Embedding), and then input into the part-of-speech tag sequence decoder to predict the part-of-speech tag in a non-autoregressive manner. The part-of-speech tag sequence decoder is a structure composed of N stacked Transformer decoder layers, each of which includes two multi-head attention modules (Multi-Head Attention), three residual normalization modules (Add&Norm) and a feedforward network layer (Feed Forward). The first multi-head attention module and the first residual normalization module perform self-attention operations; the second multi-head attention module receives the final visual representation V N After the data after the self-attention operation, the part-of-speech tag is predicted by the non-autoregressive objective function shown in formula (8)

[0090]

[0091] Where t is the time step, Ψ is the posterior network parameter, and s t For each position of the part-of-speech tag,

[0092] Generates the log-conditional likelihood of the word sequence for the decoder.

[0093] Combine Figure 1 As shown in (c), in step S400, the prior latent variable z and the posterior latent variable z~ of the visual feature are added and input into the embedding layer (Embedding), and then input into the part-of-speech tag sequence decoder to predict the part-of-speech tag in a non-autoregressive manner. The part-of-speech tag sequence decoder is a structure composed of N stacked Transformer decoder layers, each of which includes two multi-head attention modules (Multi-Head Attention), three residual normalization modules (Add&Norm) and a feedforward network layer (Feed Forward). The first multi-head attention module and the first residual normalization module perform self-attention operations; the second multi-head attention module receives the final visual representation V N After the data after the self-attention operation, the sentence is obtained by the image description optimization objective function shown in formula (9)

[0094]

[0095] in, is the data log-likelihood,

[0096] In step S500, find the optimal solution for equation (10) to ensure that L1, L2, L KL Minimum

[0097] L=L1+L2+αL KL (10)

[0098] Among them, α is the trade-off parameter between accuracy and diversity, L KL =D KL (q φ (z|s,I)||p θ (z|I)).

[0099] Furthermore, in step S500, the conditional variational evidence lower bound is obtained, and the optimal solution of formula (10) is obtained by optimizing the conditional variational evidence lower bound. Specifically:

[0100] Step S501, obtain the conditional distribution p θ (sI)

[0101]

[0102] Among them, s is the part-of-speech tag, z is the latent space;

[0103] Step S502, by maximizing the conditional distribution p θ The log-likelihood of (sI) derives the log-evidence lower bound for part-of-speech tagging

[0104]

[0105] Among them, the first term represents the log-likelihood of the data, which is used to generate the part-of-speech tag sequence, and the second term is the posterior condition and the prior condition p θ KL divergence between (zI);

[0106] Step S503, by maximizing the conditional distribution p Ψ The log-likelihood of (wI) derives a lower bound on the log-evidence of the image description.

[0107] By maximizing the conditional distribution p θ The logarithmic evidence lower bound of (sI) is obtained. This embodiment can learn a one-to-many mapping from the image space to the part-of-speech tag sequence space. It is necessary to model the image description generation process under the condition of syntactic structure embedding. That is, given an image, generate the corresponding description w m , m∈{1,2,...,M}, similar to the part-of-speech tag sequence generation, modeled as a conditional generation model p Ψ (wI)

[0108]

[0109] Where Ψ is the model parameter, the first term represents the log-likelihood of the data for generating word sequences, and the second term is the posterior parameterized by π and conditional prior p Ψ KL divergence between (zI);

[0110] To simplify the variational inference of the DCIC framework, this embodiment uses the syntactic structure embedding learned in Equation 12 as the control signal. Since each sentence corresponds to a unique part-of-speech tag sequence, assuming Therefore, Equation 13 can be rewritten as

[0111]

[0112] Optimize the loss function (Formula 10), that is, optimize Figure 1 The parameters of the neural network shown in the figure can accurately describe the image and text after input.

[0113] Examples and Comparative Examples

[0114] The image features, word embeddings, and latent vectors used in this embodiment (DCIC model) and the comparative example are all 512-dimensional. The visual encoder uses a pre-trained ResNet101 model to extract 16×16 2048-dimensional grid features of the image, and then linearly maps them to 512 dimensions. This embodiment and the comparative example use the Stanford Log-linear Part-Of-Speech Tagger to obtain the part-of-speech tags for each true label. These tags are classified into 12 categories: verbs (VERB), nouns (NOUN), adjectives (ADJ), adverbs (ADV), conjunctions (CONJ), pronouns (PRON), prepositions (ADP), particles (PRT), numeral modifiers (NUM), articles (DET), others (X), and an end marker (EOP). These part-of-speech tags are all mapped to 512-dimensional embedding vectors. In addition, the number of Transformer layers of the variational inference encoder, description, and part-of-speech tagging decoder is set to 3, and the number of attention heads is set to 8. During training, the Adam optimizer with a learning rate of 5×10^(-6) and a hyperparameter α=1.0 was used, and the model was trained for 20 rounds with a batch size of 10 under the cross entropy loss. For fairness, when evaluating the accuracy and diversity indicators of all variational inference-based models, the beam search size was set to 2 and 1 in this embodiment and the comparative example, respectively.

[0115] (1) Quantitative analysis

[0116] This embodiment and comparative example compare this method with other diverse image description methods after performing oracle reranking on the M-RNN test set. Table 1 shows the best accuracy metrics measured by different methods when sampling 20 and 100 descriptions, with the best results in bold. In terms of accuracy metrics, this embodiment significantly outperforms all competing models on all evaluation metrics, achieving the best results on the important accuracy metric CIDEr (sampling 20 descriptions: 1.693, sampling 100 descriptions: 0.000).

[0117] 1.982), demonstrating that the DCIC model can effectively leverage learned syntactic structure embeddings to improve the accuracy of diverse image descriptions. Furthermore, DCIC achieved strong results on the SPICE metric, demonstrating that the descriptions generated by DCIC are more semantically relevant to the reference sentences and more consistent with human judgment.

[0118] Table 1 Performance comparison of the accuracy of Oracle re-ranking on the MSCOCO dataset “M-RNN” test set

[0119]

[0120]

[0121] Table 2 further evaluates the diversity metric scores of the top five sentences after Consensus reordering on the MSCOCO dataset's "M-RNN" test set, as measured by different models. Table 2 shows that the DCIC model performs well in diversity metrics when sampling 20 descriptions. When sampling 100 descriptions, DCIC, limited by syntactic embedding control, can only generate sentence patterns already in the dataset. When generating a large number of descriptions, its diversity is insufficient compared to other existing models. However, compared to other comparative models, this embodiment can control sentence structure and is more flexible.

[0122] Table 2 Performance comparison of diversity after using Consensus re-ranking on the MSCOCO dataset “M-RNN” test set

[0123]

[0124] (2) Ablation experiment analysis

[0125] The effects of different training strategies on image description performance are analyzed. The different training strategies are as follows: Base: In this setting, only the image description model based on syntactic structure embedding is trained, and the part-of-speech tag sequence decoder is not trained. Base+MPSD(AT) (autoregressive translation, AT): The part-of-speech tag sequence decoder (MPSD) is trained in an autoregressive manner.

[0126] Base+MPSD(NAT) (non-autoregressive translation, NAT): Part-of-speech tag sequence decoders (MPSD) are trained in a non-autoregressive manner. Tables 3 and 4 show the accuracy and diversity ablation results of three methods on a sample of 20 descriptions. As can be seen, Base performs the worst compared to the other methods. Without MPSD, both its accuracy and diversity scores drop significantly, indicating that MPSD plays a key role in learning latent embeddings of syntactic structure. Base+MPSD(AT) significantly outperforms Base, but it performs worse in terms of controllability and sentence diversity. The Base+MPSD(NAT) model tends to generate longer captions with richer syntactic structure. More importantly, the captions generated by the Base+MPSD(NAT) model show better performance in terms of sentence pattern diversity and semantic richness of the corresponding images, indicating that the Base+MPSD(NAT) method can promote the learning of syntactic structure embeddings. Base+MPSD(NAT) also has better controllability. We then analyze the impact of the hyperparameter α on image captioning performance. Since the hyperparameter α is a trade-off between the cross-entropy loss and the KL loss, it essentially balances the diversity and accuracy of image captions. Table 5 shows the performance of the model in terms of accuracy and diversity indicators under different α values. As shown in the table, a larger α can make the model tend to generate more accurate but less diverse descriptions.

[0127] Table 3 Accuracy ablation experiments under different training strategies for “M-RNN” partitioning on the MSCOCO dataset

[0128]

[0129] Table 4 Diversity ablation experiments under “M-RNN” partitioning on MSCOCO dataset with different training strategies

[0130]

[0131] Table 5. The impact of different hyperparameter values ​​of α on performance when sampling 20 sentences in the MSCOCO dataset.

[0132]

[0133] This is because the KL loss under a larger α encourages the model to match the prior condition distribution and the posterior, which can effectively improve the accuracy of the generated description. When α is set to 0.1, the model proposed in this chapter can achieve a good trade-off between diversity and accuracy.

Claims

1. A diverse and controllable image description method based on grammatical structure embedding learning, characterized by: The following steps are involved: Step S100, obtaining visual features of image I and semantic features of text; Step S200: learning the prior approximate sub-network and the posterior approximate sub-network for the visual features and text semantic features respectively to obtain the prior latent variable z and the posterior latent variable Get the prior latent variable z and the posterior latent variable The KL divergence between Step S300, using visual features, prior latent variables z, and posterior latent variables And a series of [MASK] as input, using a non-autoregressive objective function to predict part-of-speech tags; Step S400, using visual features, prior latent variables z, and posterior latent variables Generate descriptions from text and obtain the optimization target for image description generation; Step S500, optimizing the loss function composed of the non-autoregressive objective function, the image description generation optimization objective, and the KL divergence to obtain a neural network for generating the optimal image description; The non-autoregressive objective function L1 in step S300 is Among them, t is the time step, Ψ is the posterior network parameter, s is the part-of-speech tag sequence, represents the log-likelihood of the data; In step S400, the optimization target L2 is in, represents the log-likelihood of the data, w t The word generated for step t; In step S500, the optimal solution of equation (10) is obtained to ensure that L1, L2, L KL Minimum L=L1+L2+αL KL (10) Here, α is a trade-off parameter between accuracy and diversity.

2. The method according to claim 1, characterized in that The process of learning the prior approximate sub-network in step S200 includes: Step S211: Use the multi-head attention module to N The vectors are fused into a single vector U p , U p =MHA(Q p ,V N ,V N ) (4) Among them, V N is the visual feature of image I, Q p is a learnable vector; Step S212, vector U p , fed to the feedforward network layer, and generates the prior latent variable z according to formula (5) Among them, p θ (z|I) is a priori condition, μ′ and σ′ are respectively given by FFN(U p ) to obtain the mean and standard deviation, FNN is the operation of the feedforward neural network layer, z is obtained by the re-parameterization technique From the distribution Sampling obtained.

3. The method according to claim 2, characterized in that The specific process of learning the posterior approximate sub-network in step S200 includes: Step S221, using a learnable vector Q p Learn the syntactic information from part-of-speech tags and image features and map them into a global vector U q , U q =MHA(Q q ,F q ,F q ) (6) Among them, F q is the semantic feature T q and visual features V N Convert to fusion features; F q =AddNorm(MHA(T q ,V N ,V N )) AddNorm represents the residual normalization operation, and MHA represents the operation of the multi-head attention module; Step S222, vector U q , fed to the feedforward network layer, and the posterior latent variables are generated according to formula (7) in, is the posterior condition, and They are FFN(U q ) to obtain the mean and standard deviation, By re-parameterization technique From the distribution Sampling obtained.

4. The method according to claim 1, wherein The process of obtaining the optimal solution to equation (10) in step S500 is as follows: Step S501, obtain the conditional distribution p θ (s|I) Step S502, by maximizing the conditional distribution p θ The log-likelihood of (s|I) derives the log-evidence lower bound for part-of-speech tagging Among them, the first term represents the log-likelihood of the data, which is used to generate the part-of-speech tag sequence, and the second term is the posterior condition and the prior condition p θ KL divergence between (z|I); Step S503, by maximizing the conditional distribution p Ψ The log-likelihood of (w|I) derives the log-evidence lower bound for image descriptions; Step S504: Given a picture, generate the corresponding description w m ,m∈{1,2,…M}, M represents the number of image grids, modeled as a conditional generation model p Ψ (w|I), The first term on the right side of the inequality sign represents the log-likelihood of the data, and the second term is the posterior parameterized by π. and conditional prior p Ψ The KL divergence between (z|I).

5. The method according to claim 1, wherein In step S100, the visual feature acquisition method is: For the input image I, a set of grid visual features V = {v1, v2, ..., v M }, the grid visual feature V is mapped to V 0 is fed into the standard Transformer encoder and the visual features V of the input image are obtained by formula (1) N Among them, when l=N-1, the output of the Nth attention block is obtained, that is, V l+1 =V N , MSA represents the multi-head self-attention operation, AddNorm represents the residual normalization operation, FFN represents the feedforward network layer operation, M represents the number of image grids, and d1 represents the dimension of the image feature in the latent space.

6. The method according to claim 1, characterized in that The semantic feature acquisition method is: The input text X is added through the embedding layer and position encoding as the input variable T 0 ; X is fed into the encoder, which uses a multi-head attention module and residual normalization operation on T 0 Perform self-attention operation to obtain semantic features T q T q =AddNorm(MHA(T 0 ,T 0 ,T 0 )) (2) in, T represents the length of the text, and d2 represents the dimension of the text feature in the latent space.

Images