Text image generation method based on feature fusion

Abstract

The invention discloses a text-to-image generation method based on feature fusion. The method includes the following steps that firstly, input text description information is encoded through a Bi-LSTM text encoder, sentence vector features and word vector features are generated, and coarse-grained image features are generated through a semantic space sensing module in combination with the sentence vector features and random noise; then, a text image cross-modal fusion module is used for carrying out cross-modal fusion on the generated coarse-grained image features and word vector features, so that image features with finer granularity are generated, and finally, a generator loss module containing antagonism loss, text image matching loss and perception loss is provided. And the authenticity of the image can be effectively improved. Compared with other methods, the text-to-image generation method based on feature fusion has the advantages that the effect is obviously improved, the evaluation index performance is good, and the text-to-image generation method based on feature fusion has great application value in the field of text-to-image generation.

Description

Technical Field

[0001] The present invention relates to a method for generating images from text based on feature fusion, and belongs to the field of deep learning. Background Art

[0002] The task of generating images from text is one of the most difficult tasks in the field of image generation. It belongs to the intersection of natural language processing and computer vision, and aims to generate realistic and text-consistent images from given natural language descriptions. Generating images from text can be applied to many fields such as image description generation, visual reasoning, visual question answering, and medical image generation. In recent years, with the rapid development of deep learning, the mainstream method of generating images from text has adopted generative adversarial networks.

[0003] Most existing text-to-image tasks use a stacked GAN network as the backbone network. The quality of the high-resolution image generated by this method depends on the initial image generated in the first stage. That is, if the initial image generated in the first stage is not good, the final generated image will also be poor. And because the stacked GAN introduces entanglement between different generators, the final image looks like a simple combination of blurred images and some details. To address the above problems, DFGAN and SSAGAN proposed a single-stage generative network based on a single generator and discriminator. Although DFGAN and SSAGAN have achieved good results in text-to-image tasks, they only use sentence vector features in the process of fusing text features and image features, and do not use more fine-grained word vector features, so the details of the generated image are not very perfect. In order to solve the problems existing in the single stage, cross-modal fusion of word feature vectors and image features and the introduction of perceptual loss to reduce the gap between the generated image and the target image have received much attention. Summary of the invention

[0004] The present invention proposes a method for generating images from text based on feature fusion. On the one hand, the present invention introduces word text features and constructs a text-image cross-modal fusion module to achieve cross-modal fusion of word vector features and image features, that is, the word vector features are used to guide the attention learning of image features, thereby achieving the purpose of continuously enriching image features using word vector features. On the other hand, perceptual loss is introduced to reduce the distance between the generated image features and the original image features, thereby making the generated image more realistic.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] Step 1: Input the text description into the Bi-LSTM text encoder, which encodes the text description to obtain the corresponding sentence vector feature s and word vector feature w;

[0007] Step 2: Generate coarse-grained image features by combining sentence vector features s and random noise z through the semantic space perception module;

[0008] Step 3: Use the text-image cross-modal fusion module to perform cross-modal fusion on the generated coarse-grained image features h and word vector features w to generate a more fine-grained image feature h new ;

[0009] Step 4: Use the constructed generator loss function to guide the training of the generator, so as to generate an image that is closer to the target image;

[0010] Step 5: The obtained fine-grained image features are passed through the convolution layer to finally generate a high-resolution image with a resolution of 256×256. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a framework diagram of the text generation image model based on feature fusion proposed in the present invention.

[0012] Figure 2 This is a structural diagram of the text-image cross-modal fusion module proposed in the present invention. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings:

[0014] Figure 1 This is a block diagram of the text-to-image model based on feature fusion proposed in this invention. First, the input text description information is encoded through the Bi-LSTM text encoder to generate sentence vector features and word vector features, and then the semantic space perception module combines the sentence vector features and random noise to generate coarse-grained image features.

[0015] The semantic space-aware module consists of an upsampling module, a semantic mask predictor, and a semantic space conditional batch normalization module with residual connections.

[0016] Upsampling module: Use bilinear interpolation method to expand the image feature map.

[0017] Mask predictor: The mask predictor predicts the semantic mask of the current image feature map after performing a series of convolution and batch normalization operations on the upsampled image features. It can intuitively show which parts of the current image feature map need to be enhanced by text information, so that the refined image feature map is more semantically consistent with the given text description.

[0018] Semantic spatial conditional batch normalization module: The mask feature map output by the mask predictor is used as the spatial information of semantic conditional batch normalization (SCBN) to determine the location and amount of text information added to the image feature map. The formula is as follows.

[0019]

[0020] Where m i,(h,w) represents the mask feature map features output by the mask predictor, represents the sentence vector feature, γ c (e) and represents the modulation parameters learned from the text vector features, represents the input image features, Represents the image features after batch normalization of semantic space conditions.

[0021] Finally, the image features after the sentence text information enhancement are added to the input image features to obtain new image features. The formula is as follows.

[0022]

[0023] Among them, λ 1 Represents the weight coefficient that the network automatically learns and optimizes, x nchw Represents the features after residual addition.

[0024] After the semantic space perception module combines sentence vector features and random noise to generate coarse-grained image features, the text-image cross-modal fusion module is used to cross-modally fuse the generated coarse-grained image features and word vector features to generate more fine-grained image features. Figure 2 This is the structure diagram of the text-image cross-modal fusion module proposed in this invention. Since word feature vectors and image features are two different modalities and different dimensional features, if we want to achieve text-image cross-modal fusion, we need to solve the problem of data heterogeneity between text and image. To solve this problem, the word-image alignment module is introduced. The input word feature vector is w∈R D×N , the input image feature is h∈R C×H×W , where D represents the dimension of the word feature vector, N represents the number of words, C represents the number of channels of the image feature, and H×W represents the size of the image feature map. T Multiply them together to get a new fixed-size word feature w 1 , the formula can be expressed as:

[0025] w1 =w×w T (3)

[0026] where w T ∈R N×D , w 1 ∈R D×D .

[0027] Then the word feature vector is scaled to obtain the word feature w 2 . Its formula can be expressed as:

[0028] w 2 =Respe(w 1 ) (4)

[0029] in S 0 It means D×D÷C.

[0030] Then the word feature w is transformed through the linear layer 2 Convert to new word features w 3 , the formula can be expressed as:

[0031] w 3 =Linear(w 2 ) (5)

[0032] in S 1 =H×W.

[0033] After the word feature vector and image feature are converted to the same dimension, the image feature and word feature vector are cross-modally fused. The word-image fusion module aims to use the word feature vector to guide the attention learning of image features, so as to achieve the purpose of image refinement. First, input the word feature vector size w 3 , the image feature size is The word features are mapped to the K and V feature vector spaces through the fully connected layer, and the image features are mapped to the Q feature vector space through the fully connected layer. The formula can be expressed as:

[0034] Q=Linear(h 1 ) (6)

[0035] K,V=Linear(w 3 ) (7)

[0036] The similarity probability is calculated through the Q feature vector and the K feature vector, and finally multiplied by V to get the final attention score. After the cross-modal fusion of word-image features is realized, the formula can be expressed as:

[0037]

[0038] Finally, the fused features are structurally transformed to obtain the new fused image features h 3 ∈R C×H×W , and add the residual to the previous image feature to get the final image feature h new .

[0039] h new =h+λ 2 h 3 (9)

[0040] where h new ∈R C×H×W ,λ 2 Represents the weight coefficient that the network automatically learns and optimizes.

[0041] Finally, the present invention proposes a generator loss function including adversarial loss, DAMSM loss and perceptual loss. The generator loss function is expressed as follows:

[0042]

[0043] in represents the adversarial loss, L DAMSM represents the deep attention multimodal similarity loss, L PL represents the perceptual loss, λ DA , PL Represent the weights corresponding to DAMSM loss and perceptual loss respectively. and L DAMSM It can be expressed as:

[0044]

[0045]

[0046] In tasks such as style transfer, image restoration, and super-resolution reconstruction, in order to reduce the difference in pixel space between the generated image and the original image, some methods choose to introduce perceptual loss to guide model training. Since the Resnet network model is composed of a large number of residual blocks, it can retain more feature information and will not be lost as the number of network layers increases, so a pre-trained Resnet50 network is used to extract the deep features of the generated image and the original image, and then the Euclidean distance between the two deep features is calculated and used as the perceptual loss to guide network training. Perceptual loss L PL The expression of is expressed as follows:

[0047]

[0048] Where E RResnet50 pre-trained model. represents the generated image features, and x represents the real image features.

[0049] In order to verify the effectiveness of the method of the present invention, experimental analysis was carried out on the CUB and COCO datasets. The Adam optimizer was used in the training process, where β 1 =0,β 2 = 0.9. The learning rates of the generator and discriminator are set to 0.0001 and 0.0004, respectively. Hyperparameter λ PL = 0.00005, batch_size is set to 32. Experimental comparisons are conducted with AttnGAN, DMGAN, DAE-GAN, DF-GAN, SSAGAN and other models. The experimental results are shown in Table 1.

[0050] Table 1

[0051]

[0052] From the second column of Table 1, we can see that on the CUB dataset, compared with the traditional three-stage generation network DMGAN, the IS score increased from 4.75 to 5.21, an increase of 9.9%. Secondly, for the single-stage generation networks DF-GAN and SSAGAN, their IS scores also increased from 5.10 and 5.17 to 5.21, respectively, an increase of 2.3% and 1%, respectively. The results show that the FFGAN model proposed in this paper can better improve the clarity of the generated images compared with the existing models. The third column of Table 1 lists the FID score comparison of different models on the CUB dataset. Similarly, for DMGAN, the FID score dropped from 16.09 to 13.92. And compared with DF-GAN and SSAGAN, the FID of FFGAN has also dropped significantly, indicating that the images generated by the proposed FFGAN model are closer to real images. The fourth column of Table 1 lists the FID score comparison of different models on the COCO dataset. From the results, we can see that the FID score of FFGAN is 16.99, which is 4.43 points lower than that of DFGAN and 2.38 points lower than that of SSAGAN, which also shows that the images generated by the FFGAN model are highly realistic compared to real images.

[0053] From the above comparative experiments, it can be seen that the images generated by the FFGAN model proposed in this paper are clearer and more realistic than those generated by other models.

[0054] In order to verify the effectiveness of the text image generation method based on feature fusion proposed in this paper, an ablation experiment was conducted on the CUB dataset, and the results are shown in Table 2. First, this paper adds a perceptual loss module to the baseline model SSAGAN. The results show that the IS score increased from 5.07 to 5.13, and the FID score decreased from 15.61 to 14.37, indicating that the addition of the perceptual loss module can effectively improve the clarity and realism of the model-generated pictures; then, on this basis, the text-image cross-modal fusion module is introduced, and the FID score is reduced from 14.37 to 13.54, indicating that the text-image cross-modal fusion module can effectively capture text information, so that the generated image features are more consistent with the feature distribution of real images.

[0055] Table 2

[0056]

[0057] In the SSAGAN model, the text encoder and image encoder also participate in the network training. The model is also trained based on this method in the text. The results are shown in Table 3. From Table 3, we can see that compared with the baseline model SSAGAN, the IS score increased from 5.17 to 5.21, and the FID score decreased from 16.58 to 13.92, which reflects the superiority of the model FFGAN proposed in this paper.

[0058] Table 3

[0059]

Claims

1. A method for generating images from text based on feature fusion, characterized in that The following steps are involved: Step 1: Input the text description into the Bi-LSTM text encoder, which encodes the text description to obtain the corresponding sentence vector feature s and word vector feature w; Step 2: Generate coarse-grained image features by combining sentence vector features s and random noise z through the semantic space perception module; Step 3: Use the text-image cross-modal fusion module to perform cross-modal fusion on the generated coarse-grained image features h and word vector features w to generate a more fine-grained image feature h new The principle is as follows: First, the word-image alignment module is used to input the word feature vector w∈R D×N , image feature h∈R C×H×W Convert them into the same size, and then use the word-image fusion module to map the word features to the K and V feature vector spaces through the fully connected layer, and map the image features to the Q feature vector space through the fully connected layer. Calculate the similarity probability through the Q feature vector and the K feature vector, and finally multiply them with V to get the final image feature h new ; Step 4: Use the constructed generator loss function to guide the training of the generator, so as to generate an image that is closer to the target image. The specific calculation process of the generator loss function is as follows: in represents the adversarial loss, L DAMSM represents the deep attention multimodal similarity loss, L PL represents the perceptual loss, λ DA , PL Respectively represent the weights corresponding to DAMSM loss and perceptual loss; L PL The pre-trained Resnet50 network is used to extract the deep features of the generated image and the target image, and then the Euclidean distance between the two deep features is calculated. The specific calculation process is as follows: Where E R Resnet50 pre-trained model. represents the generated image features, and x represents the real image features; Step 5: The obtained fine-grained image features are passed through the convolution layer to finally generate a high-resolution image with a resolution of 256×256.

2. The method for generating images from text based on feature fusion according to claim 1, characterized in that: The text-image cross-modal fusion module described in step three includes a word-image alignment module and a word-image fusion module, which can effectively fuse word feature vectors and image features to make the generated image more detailed.

3. The method for generating images from text based on feature fusion according to claim 1, characterized in that: The λ described in step 4 PL It is 0.00005.

Images