Fine-grained text image generation method based on single-stage generative adversarial network

Abstract

The invention provides a fine-grained text image generation method based on a single-stage generative adversarial network, relates to the technical field of computer vision, and effectively solves the problems of feature omission and fuzzy visual details in the traditional text image generation technology by constructing a generator network architecture fused with a cross-modal attention mechanism. According to the method, a semantic decoupling module is introduced in a text encoding stage, and global description vectors and local attribute features of an input text are separately encoded, especially visual sensitive elements such as colors and shapes; the image generation method not only can ensure that the content of the generated image is consistent with the semantics of the text description, but also can ensure that the generated image has more optimized image details, and increases the diversity of the generated image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a fine-grained text-to-image generation method based on a single-stage generative adversarial network. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, especially in the breakthroughs in deep learning and natural language processing, text-to-image generation technology has become one of the hotspots in computer vision and artificial intelligence research. The core goal of this technology is to convert natural language descriptions into corresponding visual images, which not only promotes the naturalness and intuitiveness of human-computer interaction, but also brings revolutionary possibilities to content creation, game development, virtual reality, etc.

[0003] In the academic field, it has become one of the most popular research directions in the field of computer vision in recent years, and has achieved remarkable results. Recurrent neural networks (RNN) and generative adversarial networks (GAN) are often combined to generate realistic images based on natural language descriptions. These methods have also provided deeper visual understanding in the industrial field of computer vision, such as virtual fitting and fashion design, personalized product customization, and product scene synthesis in advertising marketing.

[0004] Compared with text information, image information is easier to generate reading interest and easier to extract key information, with the advantages of convenient and time-saving. More and more people will first view the information presented by the image, and the text information will be placed in the second place, so the image occupies a very important position in the process of information acquisition, review, and transmission. From the current technology development, most of the complex pictures with multiple objects and detailed descriptions are still manually drawn by artists, although this way makes the image more realistic and better in detail, but it is high in cost and low in efficiency. Therefore, researchers have proposed related models for description generation, thereby automatically generating images consistent with text expressions from text descriptions, solving the problem of low efficiency and high cost in drawing, and also improving the search efficiency of users who cannot find ideal matching images under the condition of known text descriptions. Therefore, how to make the computer recognize the text information and generate an image consistent with the text description has become one of the current research focuses.

[0005] Text-to-image technology converts natural language descriptions into image content, providing new ideas and methods to solve the limitations of traditional image generation methods, realizing cross-modal mapping from natural language to visual content. This technology uses generative adversarial networks or diffusion models to analyze entity relationships in text descriptions through attention mechanisms, aligning word vectors with image latent spaces at multiple scales, and solving the bottleneck of traditional methods relying on manual feature engineering, generating single content and lacking semantic controllability.

[0006] The original GAN model includes a generator and a discriminator. The generator is optimized to generate samples similar to the real data distribution, so as to deceive the discriminator. The trained discriminator can separate the real data distribution samples from the false samples generated by the generator. The generator and the discriminator reach the optimal in the mutual game, so that the generated result is better and better.

[0007] However, when the input text involves multiple objects and relationships, the context information of the text sequence is difficult to extract, the object layout of the generated image is easy to be confused, and the details of the generated image are insufficient, which cannot meet the demand of outputting high-accuracy images. And because of the uncertainty contained in the text itself, the multi-modal information fusion between the text and the image, etc. Technical problems, it is still difficult to design a network model algorithm that can realize this function. SUMMARY

[0008] The present application provides a fine-grained text image generation method based on a single-stage generative adversarial network, which aims to solve the problem of blurred details and inconsistent semantics with text description in the generated image caused by insufficient feature fusion in traditional text image generation technology.

[0009] The present application provides a fine-grained text image generation method based on a single-stage generative adversarial network, which includes the following steps: Step S1: encode the input text through a bidirectional LSTM neural network to obtain corresponding word features and sentence features; Step S2: project and reshape the noise vector into initial image features using a fully connected layer, and then use multiple UPBlock modules to progressively upsample the initial image features to generate image feature maps; Step S3: before the last UPBlock module, input the word features and the generated feature maps into a multi-head attention module to model the word features from different angles, so as to force the generator to generate different sub-regions of the image according to the relevant words; then input the fused features into the last UPBlock module, and output the image by convolutional layer synthesis; Step S4: use the DAMSM model to extract the local region features of the word features and the output image respectively, and obtain the semantic matching loss between the image and the text according to the local region features; Step S5: the discriminator down-samples the input image into image feature maps through multiple DownBlock modules, and splices the copied sentence features and the image feature maps; Step S6: input the spliced feature map into a hybrid self-attention and convolution perception module to jointly determine the authenticity of the input image and the matching degree with the text description.

[0010] Further, in step S1, before being encoded by the bidirectional LSTM, a semantic decoupling step is further included: by a specific semantic decoupling module, the global description vector of the input text is separately encoded with the local attribute features containing color and shape attributes.

[0011] Further, in step S2, the UPBlock module is 7 in total; the dimension of the initial image feature is 4x4x512.

[0012] Further, in step S3, before the last UPBlock module, the word feature and the generated feature map are input into a multi-head attention module to model the word feature from different angles, so as to force the generator to generate different sub-regions of the image according to the specific method of the related words, which includes: The multi-head attention respectively receives the image feature generated by the sixth UPBlock module , and the word feature generated by the pre-trained text encoder ; First, the dimension of the word feature w is changed from DxL to CxL through the perception layer Q; Where D represents the word vector dimension, C is the number of channels of the image feature, and L is the number of words in the sentence; Then, the word region correlation matrix is calculated , and the correlation matrix is normalized by the softmax function, so that the attention word region feature is obtained; the specific calculation process is as follows: Where k is the number of heads used in this paper, j is the head index of the multi-head attention, i.e., the jth head, and T is the matrix transpose Transpose. In addition, the multi-head mechanism calculates the attention word region feature in multiple places, and uses element addition operation to summarize all the attention word region features in the multi-head space, and the specific calculation process is as follows: Finally, the multi-head attention feature F MH (v, w) and the image feature v are connected together to form the input feature of the next UPBlock module.

[0013] Further, in step S4, the semantic matching loss includes a fine-grained matching loss based on the similarity between the word feature and the image region feature, and a coarse-grained matching loss based on the similarity between the sentence feature and the global image feature.

[0014] Further, the DownBlock module in step S5 has 6 modules in total.

[0015] Further, the specific method of inputting the spliced feature map into the hybrid self-attention and convolution perception module in step S6 to jointly determine the authenticity of the input image and the matching degree with the text description includes: The implementation process of the convolution module is as follows: When the step is 1, a convolution kernel is used where k is the size of the convolution kernel, C in and C out are the channel sizes of the input and output respectively, and R is the real set; since the input tensor and the output tensor are and , the convolution is: where, In , p, q represent kernel weights, ranging from {0, 1,..., k-1}, is a value extracted from the input feature map f according to the current position (p, q) of the convolution kernel and the output position (i, j) of the corresponding coordinates; Then, by introducing a shift operation, it is simplified as: where: where, is the output feature map after shifting, Shift(·) is the shift function, Δx represents the displacement in the x-axis (horizontal direction), and Δy represents the displacement in the y-axis (vertical direction); For all i, j, we have: Therefore, the convolution process includes two stages: Stage I: ; Stage II: ; Finally, we get ; The implementation process of the self-attention module is as follows: in the standard self-attention module, there are N heads, and the output is calculated as follows: where, represents the concatenation of the outputs of the N heads, is the projection matrix of the query, key and value, and A represents the attention weight function, which is responsible for calculating the correlation score between two positions; The attention weight calculation in the common self-attention module is as follows: where d is the feature dimension; this process is also divided into two stages: Stage I: ; Stage II: ; The implementation process of the hybrid self-attention module is as follows: The integration of convolution and self-attention includes the following two steps: Step (1): the input features are projected and reshaped into N segments using three 1x1 convolutions, thereby generating 3xN intermediate feature maps; In step (2), the intermediate features are used in different ways; for the self-attention part, the intermediate features are divided into N sets, where each set is composed of three feature maps obtained by three 1x1 convolutions; In addition, the outputs of the two paths are combined by addition, and their strengths are controlled by two learnable scalars and The formula is as follows: where represents the output feature map of the self-attention path, represents the output feature map of the convolution path, is a learnable scalar weight for controlling the contribution degree of the self-attention path to the final output, is a learnable scalar weight for controlling the contribution degree of the convolution path to the final output, represents the final output feature map after fusion.

[0016] Compared with the prior art, the present application has the following beneficial effects: The present application introduces a multi-head attention module to simulate different aspects of word information in order to generate images that are more consistent with the text description; hybrid self-attention and convolution are integrated into the discriminator to ensure that the generated images closely match the provided text description; therefore, the discriminator encourages the generator to selectively focus on relevant image areas, ensuring the integrity and diversity of the generated images.

[0017] On the basis of the implementation manners provided in the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which a few embodiments of the present application are shown by way of illustration. In the drawings: Figure 1 is a whole framework diagram of the method of the present application; Figure 2 is a multi-head attention structure diagram of the present application; Figure 3 is a convolution module structure diagram of the present application; Figure 4 is a self-attention module structure diagram of the present application; Figure 5 is a module structure diagram of the present application integrating self-attention and convolution. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present application disclosed herein will now be described in detail with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art. Technical means used in the examples are conventional means well known to those skilled in the art, unless otherwise specified.

[0020] The present application provides a fine-grained text-to-image generation method based on a single-stage generative adversarial network, as shown in Figures 1 to 5 comprising the following steps: 1) encode the input text through a bidirectional LSTM neural network to obtain relevant word features and sentence features.

[0021] 2) project the noise vector z to the visual domain using a fully connected layer, then reshape it to obtain a 4x4x512 image feature vector, and then apply 7 UPBlock modules to upsample the image features.

[0022] 3) insert the proposed multi-head attention module before the last UPBlock module to model the word features from different angles, thereby forcing the generator to draw different sub-regions of the image based on the most relevant words. The synthesized image is converted by a convolution layer that collects image features from the feature map.

[0023] 4) use the DAMSM model to calculate the image-text matching loss, which has two inputs: the word features generated by the bidirectional LSTM neural network and the local image features of the synthesized image extracted by the image encoder.

[0024] 5) The discriminator uses 6 Down-Block modules to convert the image into image feature maps, and then appends the copied sentence vector to the image features.

[0025] 6) The result obtained in step 5) is input into the hybrid self-attention and convolution perception module to further determine whether the image is true or false and whether the image and text match.

[0026] In this embodiment, noise vectors are sampled from Gaussian distribution to maintain the diversity of generated images. The proposed multi-head attention module is inserted before the last UPBlock module, which will model the word information from different angles, forcing the generator to draw different sub-regions of the image according to the most relevant words.

[0027] Generating a sub-region of an image may be associated with multiple texts from different angles, so the attention mechanism and the multi-head mechanism of the encoder in the Transformer are mixed together, so that different heads will consider the words related to the sub-region of the image from different angles. Then, our proposed multi-head attention mechanism summarizes the information under multiple heads, comprehensively collecting relevant word information when generating a sub-region.

[0028] As shown in Figure 2 , the specific implementation method of the multi-head attention module proposed in step 3) is as follows: The multi-head attention respectively receives the image features generated by the sixth UPBlock module, and the word features generated by the pre-trained text encoder. First, the dimension of w is changed from D x L to C x L through the perception layer Q. Next, the word region correlation matrix is calculated , and the correlation matrix is normalized through the softmax function, so that the attention word region feature is obtained. The specific calculation process is as follows.

[0029] Where k is the number of heads used in this paper.

[0030] In addition, the multi-head mechanism calculates the attention word region feature in multiple places, and uses element addition operation to summarize all the attention word region features in these multi-head spaces, and the specific calculation process is as follows.

[0031] Finally, the multi-head attention feature F MH (v, w) and the image feature v are connected together to form the input feature of the next UPBlock module.

[0032] While traditional attention mechanisms may focus on the most relevant words when creating sub-regions of an image—a word often has different meanings depending on the context—our proposed multi-head attention mechanism comprehensively considers more word information, making the generated images more consistent with the text description. Furthermore, a series of UPBlocks modules progressively upsample the synthesized image. Therefore, we apply the multi-head attention module only before the last UPBlock module to utilize word-level information for the final fine-grained image generation.

[0033] The specific implementation method of the hybrid self-attention and convolutional perception module proposed in step 6) is as follows: like Figure 3 As shown, the implementation process of the convolution module is as follows: When the stride is 1, use one convolutional kernel. Where k is the size of the convolution kernel, C in and C out These are the channel sizes for the input and output, respectively. Since the input and output tensors are... and Then the convolution is: in exist In this context, p and q represent kernel weights, ranging from {0, 1, ..., k-1}.

[0034] This can be simplified by introducing a shift operation. ,in: For all i and j, we get: Therefore, the convolution process consists of two stages: Phase I: , Phase II: , The final result is: , The implementation process of the self-attention module is as follows: In a standard self-attention module, there are N heads, and the output is calculated as follows: in This represents the concatenation of N header outputs. It is a projection matrix of queries, keys, and values.

[0035] like Figure 4 As shown, the attention weights in common self-attention modules are calculated as follows: where d is the feature dimension. This process is also divided into two stages: Stage I: Stage II: .

[0036] As shown in Figure 5 , the implementation process of the mixed self-attention module is as follows: the integration of convolution and self-attention includes two steps. In step I, the input features are projected and reshaped into N segments using three 1x1 convolutions, resulting in 3xN intermediate feature maps. In step II, the intermediate features are used in different ways. For the self-attention part, the intermediate features are divided into N sets, each consisting of three feature maps obtained by three 1x1 convolutions. In addition, the outputs of the two paths are combined by addition, and their strengths are controlled by two learnable scalars and , as follows.

[0037] where represents the output feature map of the self-attention path, represents the output feature map of the convolution path, is a learnable scalar weight that controls the degree of contribution of the self-attention path to the final output, is a learnable scalar weight that controls the degree of contribution of the convolution path to the final output, represents the final output feature map after fusion.

[0038] One-way discriminators are used to optimize the discriminators in the network. These discriminators are trained using adversarial loss and gradient penalty loss.

[0039] To verify the effectiveness and reasonableness of the algorithm proposed in the present application, the CUB-200-2011 bird dataset is used for training and testing under the same experimental environment and conditions. The dataset contains 200 bird species, with a total of 11,788 images. These data are divided into 8,855 training images and 2,933 test images. Each image is accompanied by ten different text descriptions.

[0040] The present application follows previous work and adopts widely used Inception Score (IS), Fréchet Inception Distance (FID) to quantify the performance. For Inception Score, a pre-trained Inceptionv3 network is used to compute the KL divergence between the conditional class distribution (generated images) and the marginal class distribution (real images). A larger IS indicates that the generated images have higher quality, and each image is obviously a specific class. Fréchet Inception Distance (FID) calculates the Fréchet distance between the feature distributions of generated images and real images. These features are extracted by a pre-trained Inceptionv3 network. A lower FID means that the generated images are more realistic, and the quantitative comparison results with multiple current state-of-the-art text-to-image (T2I) models are shown in Table 1.

[0041] Table 1 Experimental comparison Method IS↑ FID↓ AttnGAN 4.32 26.08 StackGAN 3.77 29.65 StackGAN++ 3.86 25.49 DF-GAN 4.69 19.62 SSA-GAN 4.78 19.21 The invention 4.98 17.93 As shown in Table 1, the quantitative comparison experimental results fully show that the method proposed in the present application achieves the current leading performance on the CUB-200-2011 dataset. Specifically: In terms of image quality (IS): the IS score of the present application is 4.98, which is significantly higher than that of other comparative models (such as 4.32 of AttnGAN and 4.69 of DF-GAN). A higher IS score indicates that the generated images of the present application not only have higher visual quality, but also have better semantic consistency between the content and the text description, and the generated images have more clear class distinction.

[0042] In terms of image realism (FID): the FID score of the present application is the lowest, which is 17.93, much lower than that of other comparative models. A lower FID score proves that the generated images of the present application are closer to real images in feature distribution, that is, the generated images are more realistic, natural, and have more details.

[0043] In summary, by introducing semantic decoupling encoding, multi-head attention mechanism in the generator, and hybrid self-attention and convolution modules in the discriminator, the present application effectively improves the semantic alignment accuracy and visual detail quality of the generated images and the text description. Experimental data strongly verify the effectiveness of the present application in solving the technical problem of "feature omission and visual detail ambiguity", and achieve the expected beneficial effects.

[0044] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A fine-grained text-to-image generation method based on a single-stage generative adversarial network, characterized in that, Includes the following steps: Step S1: Encode the input text using a bidirectional LSTM neural network to obtain the corresponding word features and sentence features; Step S2: Project the noise vector using a fully connected layer and reshape it into initial image features. Then, use multiple UPBlock modules to upsample the initial image features step by step to generate an image feature map. Step S3: Before the last UPBlock module, the word features and the generated feature map are input into the multi-head attention module to model the word features from different angles, so as to force the generator to generate different sub-regions of the image according to the relevant words; then the fused features are input into the last UPBlock module, and the output image is synthesized by the convolutional layer. Step S4: Using the DAMSM model, extract the word features and the local region features of the output image respectively, and obtain the semantic matching loss between the image and the text accordingly; Step S5: The discriminator downsamples the input image into an image feature map through multiple DownBlock modules, and concatenates the copied sentence features with the image feature map; Step S6: Input the stitched feature map into the hybrid self-attention and convolutional perception module to jointly determine the authenticity of the input image and its degree of matching with the text description.

2. The fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, In step S1, before bidirectional LSTM encoding, a semantic decoupling step is also included: through a specific semantic decoupling module, the global description vector of the input text is separately encoded with the local attribute features containing color and shape attributes.

3. The fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, There are a total of 7 UPBlock modules mentioned in step S2; The initial image features have a dimension of 4×4×512.

4. The fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, The specific method described in step S3, which involves inputting the word features and the generated feature map into the multi-head attention module before the last UPBlock module to model the word features from different angles and force the generator to generate different sub-regions of the image based on the relevant words, includes: Multi-head attention receives image features generated by the sixth UPBlock module. and word features generated by a pre-trained text encoder. ; First, the dimension of the word feature w is changed from D×L to C×L through the perceptual layer Q; Where D represents the word vector dimension, C is the number of channels of the image feature, and L is the number of words in the sentence; Then through Calculate the word region relevance matrix The correlation matrix is ​​then normalized using the softmax function, thus obtaining the attention word region features. The specific calculation process is as follows: Where k is the number of heads used in this paper, j is the head index of multi-head attention, i.e. the j-th head, and T is matrix transpose; Furthermore, the multi-head mechanism computes attention word region features in multiple locations and uses element-addition operations to summarize all attention word region features in these multi-head spaces. The specific computation process is as follows: Finally, the multi-head attention feature F MH (v, w) and the image feature v are connected together to form the input features of the next UPBlock module.

5. A fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, In step S4, the semantic matching loss includes fine-grained matching loss based on the similarity between word features and image region features, and coarse-grained matching loss based on the similarity between sentence features and global image features.

6. The fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, There are a total of 6 DownBlock modules mentioned in step S5.

7. A fine-grained text-to-image generation method based on a single-stage generative adversarial network according to claim 1, characterized in that, The specific method for inputting the concatenated feature map into the hybrid self-attention and convolutional perception module in step S6 to jointly determine the authenticity of the input image and its degree of matching with the text description includes: The implementation process of the convolution module is as follows: When the stride is 1, use one convolutional kernel. Where k is the size of the convolution kernel, C in and C out These are the channel sizes of the input and output, respectively, and R is the set of real numbers; since the input and output tensors are... and Then the convolution is: in, exist In this context, p and q represent kernel weights, ranging from {0, 1, ..., k-1}. This represents the value extracted from the input feature map f at the corresponding coordinates calculated based on the current position (p,q) of the convolution kernel and the output position (i,j). Then, we simplify this by introducing a shift operation: ,in: in, It is the output feature map after shifting. Shift(·) is the shift function. Δx represents the displacement on the x-axis (horizontal direction) and Δy represents the displacement on the y-axis (vertical direction). For all i and j, we get: Therefore, the convolution process consists of two stages: Phase I: ; Phase II: ; Finally obtained ; The implementation process of the self-attention module is as follows: In a standard self-attention module, there are N heads, and the output is calculated as follows: in, This represents the concatenation of N header outputs. It is the projection matrix of query, key and value, and A represents the attention weight function, which is responsible for calculating the relevance score between two positions; The attention weights in common self-attention modules are calculated as follows: Where d is the feature dimension; this process is also divided into two stages: Phase I: ; Phase II: ; The implementation process of the hybrid self-attention module is as follows: The integration of convolution and self-attention involves the following two steps: Step (1): The input features are projected using three 1×1 convolutions and reshaped into N segments, resulting in 3×N intermediate feature maps; In step (2), the intermediate features are used in different ways; for the self-attention part, the intermediate features are divided into N sets, where each set consists of three feature maps obtained by three 1×1 convolutions; Furthermore, the outputs of the two paths are combined additively, and their strength is determined by two learnable scalars. and Control, the formula is as follows: in, The output feature map represents the self-attention path. This represents the output feature map of the convolution path. It is a learnable scalar weight used to control the self-attention path. The degree of contribution to the final output, It is a learnable scalar weight used to control the convolution path. The degree of contribution to the final output, F out This represents the final output feature map after fusion.

Images