A method for generating images from text based on cross-attention encoding

Abstract

The present invention discloses a method for generating images from text based on cross-attention coding. The method comprises the following steps: the text description is passed through a cross-attention encoder to obtain a cross-attention feature vector and a word feature matrix. The cross-attention encoder is a network that needs to be pre-trained. The network can extract text information and original image information, and translate and align these two cross-modal information, and finally output a cross-attention feature vector after joint encoding of language and vision. The cross-attention feature vector is input into a classic three-level adversarial generative network, and realistic images with resolutions of 64×64, 128×128, and 256×256 are generated step by step. The method for generating images from text based on cross-attention coding described in the present invention is significantly improved over other methods, and the comprehensive performance of evaluation indicators is good, which can basically meet the needs of general tasks of generating images from text.

Description

Technical Field

[0001] The present invention designs a method for generating images from text based on cross-attention coding, which relates to the fields of deep learning, computer vision technology and natural language processing. Background Art

[0002] With the development of generative adversarial networks (GANs), a number of tasks have emerged across the fields of computer vision and natural language processing, including image description generation, sketch coloring, visual question answering, and text-to-image synthesis. Generating images from text descriptions has been a key subtask studied by scholars in both fields in recent years. The key to this task lies in building a bridge between the two different modalities of language and image information, enabling network models to generate high-quality, realistic images that match the text descriptions. Because generative adversarial networks demonstrate excellent performance in handling multimodal problems, GANs are widely used in this research task.

[0003] In recent years, the leading approaches to generating images from text have first encoded the text description and then used the encoded text description as additional conditional information for the generator, using single- or three-stage GANs to generate realistic images. Single-stage GANs generate low-resolution images, while three-stage GANs progressively generate high-resolution images with resolutions of 64×64, 128×128, and 256×256. However, current mainstream text-to-image network models directly encode the text description using a pre-trained text encoder. This independent encoding of the text description fails to consider the semantic connection between the text and the corresponding image information, ignoring the semantic gap between the language and image spaces. This results in a poor match between the generated image and the text, thus affecting the quality of the generated image. Among GAN-based text-to-image network models, the dynamic memory generative adversarial network (DM-GAN) model is a very representative one, but its pre-trained text encoder still ignores the semantic gap between the language and image spaces.

[0004] Recently, there have been major breakthroughs in the study of joint encoding representations of language and vision based on Transformer. This cross-modal cross-attention mechanism aims to capture the mapping relationship between language and vision, and can effectively build a communication bridge between these two cross-modal information, thereby effectively improving the performance of vision-language cross-task models. Summary of the Invention

[0005] To address these issues, the present invention provides a method for generating images from text based on cross-attention encoding. This method introduces a cross-attention mechanism into a pre-trained text encoder to capture the inherent connection between language and image information, thereby encoding text descriptions more accurately.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] A method for generating images from text based on crisscross attention encoding, comprising the following steps:

[0008] (1) The text description is input into the cross-attention encoder, which jointly encodes the text description and outputs a cross-attention feature vector f c and a word feature matrix W.

[0009] (2) Cross-attention feature vector f c Combined with the noise z to form the feature vector f c' , f c' Input the primary generation network to generate a 64×64 low-resolution initial image with the initial image feature F0.

[0010] (3) Input F0 and the simple feature matrix W into the dynamic memory module. The dynamic memory module will select the corresponding word information to optimize the image feature F0 to obtain F0'.

[0011] (4) Feature F0' is the input of the secondary generation network. The secondary generation network generates an image with a resolution of 128×128, and the image feature is F1. After optimization by the dynamic storage module, feature F1' is obtained.

[0012] (5) Feature F1' is input to the three-stage generation network. Finally, the three-stage generation network generates a high-resolution image with a resolution of 256×256. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the cross attention encoder model diagram.

[0014] Figure 2 This is a block diagram of the text generation image model based on the cross-attention encoder. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings:

[0016] Figure 1 This is the cross attention encoder model diagram:

[0017] The cross attention encoder is a module that needs to be pre-trained for different data sets. In the pre-training stage, the original image and the corresponding text description are input into the cross attention encoder, which outputs a cross attention feature vector f c And a word feature matrix W, the detailed process is as follows:

[0018] (1) Input the original image into the image feature extraction network to obtain the original image feature vector f v , the image feature extraction network uses the InceptionV3 network.

[0019] (2) The corresponding text description is input into the text feature extraction network to obtain a global sentence vector s and a word feature matrix W. The text feature extraction network uses a bidirectional LSTM network.

[0020] (3) The global sentence vector s and the image feature vector f v Mapped to two feature spaces k through two linear layers respectively s ,v s ,q s and k v ,v v ,q v , calculate the cross attention score.

[0021]

[0022] (4) The score is normalized by the Softmax function to obtain the weight S c , consisting of weights and feature vectors v s Multiply to get the feature vector l, l is regularized to get the cross-coding l c .

[0023] S c =Softm(score)

[0024] l=s c ·v s

[0025] l c =Normalization(A1l+B1)

[0026] (5) The cross-coding l c The self-attention network is used to further retrieve context information. The process is as follows: c After the linear layer is mapped to the feature space q l ,k l ,v l , calculate the self-attention weight S s , by weight and feature vector v l Multiply to get the eigenvector lcs , finally, l cs After regularization, we get the cross attention feature vector f c .

[0027] q l ,k l ,v l =Linear(l c )

[0028]

[0029] l cs =s s ·v l

[0030] f c =Normalization(A2l cs +B2)

[0031] Figure 2 This is a block diagram of the text-to-image model based on the cross-attention encoder, which includes the following steps:

[0032] (1) Input the text description into the cross-attention encoder to obtain the cross-attention feature vector f c and word feature matrix W,f c After combining with Gaussian noise, the initial image feature F0 is input into the primary generation network. The initial image feature generates an initial image with a resolution of 64×64 through the primary generator G0.

[0033] (2) F0 and the word feature matrix W are input into the dynamic memory module, which selects word information to update the initial image feature F0 to F0'. F0' is input into the second-level generation network to generate the second-level image feature F1. F1 generates an image with a resolution of 128×128 through the second-level generator G1.

[0034] (3) Similarly, F1 is updated to F1' through the dynamic storage module, and F1' is input into the third-level generation network to obtain the generated image feature F2. F2 is generated by the generator G2 to generate the final realistic image with a resolution of 256×256.

[0035] The CUB-200-2011 bird dataset has 8,855 images in its training set and 2,933 images in its test set. Experiments were conducted on the CUB dataset. First, a cross-attention encoder was pre-trained with a batch size of 48 and a learning rate of 0.002 for 250 epochs. The saved pre-trained encoder model was then loaded to train a separate generative adversarial network with a batch size of 20 and a learning rate of 0.0002 for 1,800 epochs. The experimental results of the present invention are shown in Tables 1 and 2. The images generated by the present invention were tested for their IS (Inception Score) and FID (Fréchet Inception Distance) scores. A higher IS score indicates better clarity and greater diversity in the generated image; a lower FID score indicates a closer, more realistic image.

[0036] Table 1 IS scores of different models on the CUB dataset

[0037]

[0038] Table 2 FID scores of different models on the CUB dataset

[0039]

[0040] As shown in Table 1, compared with other models, the IS score of the present invention is significantly improved. As shown in Table 2, the FID score of the present invention is significantly reduced, which verifies the effectiveness of the present invention.

Claims

1. A method for generating images from text based on cross-attention encoding, characterized by The following steps are involved: Step 1: Input the text description into the cross-attention encoder, which inputs the corresponding text description into the text feature extraction network to obtain a global sentence vector and a word feature matrix. The global sentence vector and the image feature vector are mapped to two feature spaces and respectively through two linear layers, and the cross-attention score is calculated; the obtained weight and feature vector are multiplied by normalization, and the cross-coding is obtained after regularization; the cross-coding is further retrieved through the self-attention network to finally output a cross-attention feature vector f c and a word feature matrix W; Step 2: Cross-attention feature vector f c After being combined with Gaussian noise z, it is input into the primary generation network to generate a 64×64 low-resolution initial image with the initial image feature F0; Step 3: Input F0 and word feature matrix W into the dynamic memory module. The dynamic memory module will select the corresponding word information to optimize the image feature F0 to obtain F0'; Step 4: Feature F0' is input into the second-level generative network to generate an image with a resolution of 128×128. The image feature is F1. F1 is optimized by the dynamic storage module to obtain feature F1'. Step 5: Feature F1' is input into the third-level generation network to finally generate a high-resolution image with a resolution of 256×256.

2. The method for generating images from text based on cross-attention coding according to claim 1, characterized in that The cross-attention encoder in step 1 is composed of four parts: text feature extraction, image feature extraction, cross-attention encoding, and self-attention encoding. The text feature extraction module is implemented based on a bidirectional long short-term memory network BiLSTM, which extracts text information and outputs a word feature matrix W and a global sentence feature vector s. The image feature extraction module is implemented based on the InceptionV3 network to extract the original image features f v The global sentence feature vector s is aligned and translated with the original image features by the cross-attention encoding module, and then further retrieved by the self-attention encoding module to update the cross-attention feature vector f c .

3. The method for generating images from text based on cross-attention coding according to claim 1, characterized in that The cross attention feature vector f in step 2 c It is proposed based on the idea of cross-modal encoding representation of Transformer, c It does not simply represent text description information, f c It is a joint encoding that captures the internal connections between text information and image information.

4. The method for generating images from text based on cross-attention coding according to claim 2, characterized in that The cross-attention encoding module in step 2 is used to build the intrinsic connection between text features and image features, aiming to calculate the global sentence feature vector s and the original image feature f v The cross attention score is obtained to obtain the cross attention weight, thereby optimizing the global sentence features. The specific calculation method is as follows: q s ,k s ,v s =Linear(s) q v ,k v ,v v =Linear(f v ) score'=Soft(score) s c =dropout(score') l=s c ·v s l c =Normalization(A1l+B1) In the above formula, q s ,k s ,v s It is obtained by mapping the text feature s to a feature space through a linear layer, q v ,k v ,v v is the image feature f v It is obtained by mapping the linear layer to a feature space. Linear represents the linear layer. The two linear layers respectively convert s and f v Mapped to two feature spaces, λ c is a custom constant, soft represents the softmax function, dropout function prevents overfitting, Normalization function represents regularization processing, l c is the joint encoding output by this module, and A1 and B1 are the parameters to be learned.

5. The method for generating images from text based on cross-attention coding according to claim 2, characterized in that The self-attention encoding module in step 2 can further retrieve the text description context information to calculate the self-attention score and obtain the self-attention weight, thereby further optimizing the encoding joint encoding l c , the specific calculation process is as follows: q l ,k l ,v l =Linear(l c ) l cs =s s ·v l <h2 style=";text-align:left;direction:ltr">f<h2 style=";text-align:left;direction:ltr"> c <h2 style=";text-align:left;direction:ltr"> =A2l<h2 style=";text-align:left;direction:ltr"> cs <h2 style=";text-align:left;direction:ltr"> +B2 In the above formula, q l ,k l ,v l Yes c Obtained by mapping a linear layer to a feature space, λ s It is a custom constant. Linear represents the linear layer, Soft represents the softmax function, and the dropout function prevents overfitting. A2 and B2 are learned through training.

Images