Text-to-image synthesis method based on efficient attention generative adversarial network

Abstract

The invention provides a text-to-image synthesis method based on an efficient attention generative adversarial network. According to the method, high-efficiency image synthesis is realized by adopting a single-stage generative adversarial network model, and an efficient attention mechanism is combined to optimize the layout structure of the synthesized image. The mechanism can dynamically adjust the weights of different regions of the image, gives consideration to the transmission and communication of image features and the updating and integration of text features, solves the problems that the quality of a synthetic image of a GAN (Generative Adversarial Network) in the prior art is too dependent on the quality of an initial low-resolution image, and a deviation exists between a finished image and original text description semantically, and improves the image quality. Compared with the prior art, the method has the advantages that the image synthesis is obviously enhanced in the aspects of diversity and semantic consistency, and meanwhile, the feature map is finely adjusted and refined through fusion, iteration and convolution of the feature map, so that the final high-resolution and realistic image is generated, and the influence of the initial image quality on the quality of the synthesized image is eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of text-to-image synthesis, and in particular to a text-to-image synthesis method based on an efficient attention-generated adversarial network. Background Art

[0002] In the field of image processing, the continuous progress and application of deep learning technology has promoted the development of text-to-image synthesis technology, making it one of the current research hotspots. The goal of text-to-image synthesis technology is to convert natural language text into corresponding visual images. However, due to the complexity of the model structure, generating images with both high quality and strong semantic consistency is a challenging task. Text-to-image synthesis technology not only brings new research directions to the field of computer science, but also creates innovation and development opportunities in many fields such as human-computer interaction, education, and entertainment. Text-to-image synthesis technology has also been widely used in multiple image synthesis tasks in the fields of image editing, face synthesis, image restoration, and image super-resolution.

[0003] At present, generative adversarial networks (GANs) have achieved remarkable success in the field of image processing and have received extensive attention and research. As a deep learning model, the core idea of ​​generative adversarial networks is to train two deep neural networks: the generator and the discriminator. In the GAN model, the goal of the generator is to generate realistic data samples from random noise, while the discriminator is responsible for distinguishing whether the input data comes from a real data set or fake data generated by the generator. During the training process, the two networks interact with each other through an adversarial game mechanism, where the generator strives to generate data that can deceive the discriminator, while the discriminator strives to improve its ability to identify fake data. This interaction ultimately leads to the generator being able to synthesize samples that are difficult to distinguish from real data. The development of this technology has brought revolutionary changes to the field of text-to-image synthesis and opened up new possibilities for a variety of application scenarios.

[0004] The existing text-to-image synthesis methods usually use the stacked structure of generative adversarial networks to perform image synthesis, and generally follow a synthesis strategy from coarse to fine. This method usually synthesizes a low-resolution initial image first, and then gradually improves the image resolution through an optimization process, and finally obtains a high-resolution and accurate image. However, the limitation of this method is that there is a lack of effective correlation between the text features used in the synthesis process at each stage, resulting in the image synthesized in the later stage mainly focusing on the transmission and communication of image features, while ignoring the update and integration of text features. This processing method may cause the synthesized image to be semantically deviated from the original text description, thereby affecting the final synthesis effect; in addition, the quality of the synthesized image by this method depends largely on the quality of the initial low-resolution image. Summary of the invention

[0005] The present invention provides a text-to-image synthesis method based on an efficient attention-generated adversarial network, which solves the problems in the prior art that the quality of images synthesized by a generative adversarial network (GAN) is too dependent on the quality of an initial low-resolution image, and that the synthesized image has a semantic deviation from the original text description.

[0006] Technical solution:

[0007] To achieve the above object, the present invention adopts the following technical solution: a text-to-image synthesis method based on an efficient attention-generated adversarial network, comprising the following steps:

[0008] Step 1: Preprocess the given text and convert each word into a vector of fixed size;

[0009] Step 2: Input the fixed-size word vector in step 1 into the Bi-LSTM, extract the word vector W and sentence vector S from the Bi-LSTM output, and input them into the generator;

[0010] Step 3: Input W and S generated by Bi-LSTM into the generator, and input the noise vector z sampled from the Gaussian distribution, and map the noise vector z to the high-dimensional feature space through the fully connected layer;

[0011] Step 4: Set the upsampling ratio to 2 and use a bilinear interpolation upsampling layer to convert the latent space vector into a 4×4 feature map F.

[0012] Step 5: Input the feature map F in step 4 into the diversity injection module to enhance the diversity of image features;

[0013] Step 6: Iterate steps 4 and 5 three times to obtain a feature map F′ that is gradually enlarged from 4×4 to 32×32;

[0014] Step 7: Input the feature map F′ into the text enhancement module to enhance the semantic consistency of the text;

[0015] Step 8: Fuse the feature maps output by step 7 and step 6, and then iterate the fused feature map 3 times to obtain a feature map F″ with a size of 256×256;

[0016] Step 9: Perform a convolution operation on the feature map F″ obtained in step 8 to generate the final composite image.

[0017] Furthermore, in step 1, the given text is segmented into individual words and a vocabulary is constructed such that each unique word is mapped to a unique index, and each word is converted into a fixed-size vector by looking up a pre-trained word vector matrix.

[0018] Furthermore, in step 2, the forward LSTM processes from the beginning to the end of the sequence, and the reverse LSTM processes from the end to the beginning of the sequence. At each time step, the hidden states of the forward and reverse LSTMs are merged to generate a word vector W containing forward and backward information. After the Bi-LSTM processes the entire sequence, the hidden state of the last time step is taken as the sentence vector S as the input of the generator.

[0019] Furthermore, in step three, the process of mapping the noise vector z to the high-dimensional feature space through the fully connected layer includes the following steps:

[0020] (1) Perform a linear transformation on the noise vector z. The transformation is defined by a weight matrix w and a bias vector b. The dimension of the weight matrix w is m×n, where m represents the output dimension of the fully connected layer and n represents the dimension of the input noise vector z. The mathematical expression of the linear transformation is:

[0021] Z=w×z+b

[0022] Where Z is the intermediate result obtained after linear transformation;

[0023] (2) The intermediate result Z is processed by the nonlinear activation function f to obtain the output:

[0024] Y = f(Z).

[0025] Furthermore, in step 4, the use of a bilinear interpolation upsampling layer to convert the latent space vector into a feature map F of size 4×4 includes the following steps:

[0026] (1) Determine the corresponding position of each pixel Fij in the 4×4 feature map F in the 2×2 feature map, calculate the distance between the pixel and the nearest four pixels, and determine the weight. Specifically, the coordinates (i, j) of the 4×4 feature map are normalized to the coordinate range of the 2×2 feature map, that is, the normalized row coordinates and column coordinates are divided by 2 respectively to obtain the normalized coordinates (r, c), which are calculated as follows: Normalized row coordinates: r = i / 2; Normalized column coordinates: c = j / 2; where i and j are the row and column indices of the pixel in the 4×4 feature map (i, j∈[0, 3]). By taking the integer part of the normalized coordinates, that is, using the floor function, the position of the main pixel in the 2×2 feature map (r int , C int ), the calculation formula is: The row index of the main pixel: r int =floor(r); column index of the primary pixel: C int =floor(c). Next, we will calculate the normalized coordinates (r, c) and the rounded coordinates (r int , C int), which represents the distance between the pixel point Fij and the nearest four pixels in the 2×2 feature map. The calculation formula is: Horizontal distance: dx = rr int ; Vertical distance: dy = cC int . Afterwards, these distances are used to calculate weights, and the calculation formula is: weight v00 = (1-dx)*(1-dy); weight v01 = dx*(1-dy); weight v10 = (1-dx)*dy; weight v11 = dx*dy.

[0027] (2) Perform a weighted average of the values ​​of the four nearest pixels according to the weights in step (1) to obtain a new pixel value. Taking F00 as an example, its pixel value is calculated as follows: F00 = weight v00*v00 + weight v01*v01 + weight v10*v10 + weight v11*v11.

[0028] (3) Repeat steps (1) and (2) until all 16 pixels in the 4×4 feature map are calculated. The specific steps are: starting from F00, calculate the value of each pixel in turn until F33, and we get the complete 4×4 feature map F.

[0029] Furthermore, in step 5, the step of inputting the feature map F into the diversity injection module includes the following steps:

[0030] (1) First, the feature map F obtained in step 4 and the sentence vector S obtained in step 2 are input into the feature fusion layer of the diversity injection module for fusion processing. The calculation formula is:

[0031] F′ c =γ c (S)×F c +β c (S)

[0032] In the formula, γ c (S) and β c (S) are the scale parameter and bias parameter based on the sentence vector S, respectively, and c represents each channel in the feature map F;

[0033] (2) After processing, the feature map F′ c The noise N is injected into the , and the calculation formula of the injection operation is:

[0034] F N =F′ c +σN

[0035] Among them, σ is a learning variable, which is set to 0 at the initial stage of the training process, and F N Represents the image features after noise processing.

[0036] (3) The feature map F output in step (2) after the noise N is injected N Input into a 3×3 convolutional layer to capture the texture and structure information of the local area;

[0037] (4) The image features and sentence vector S are further processed through the feature fusion layer, and the output image features are sent to the self-attention mechanism to capture the long-range dependencies in the image and coordinate the generated local image features with the global context. In the self-attention mechanism, for each position in the feature map, its query vector is dot-producted with all key vectors to obtain an attention weight matrix SA, which is calculated as follows:

[0038]

[0039] Where Q is the set of all query vectors, K is the set of all key vectors, and d k is the dimension of the key vector, and the softmax function is used to normalize the weights;

[0040] (5) Concatenate the weighted summation result with the original feature map F to obtain a new feature map FS A .

[0041] Furthermore, in step six, in each iteration, the size of the feature map is enlarged to twice its original size through an upsampling operation.

[0042] Furthermore, in step 7, the step of inputting the feature graph F′ into the text enhancement module to enhance the text semantic consistency specifically includes the following steps:

[0043] (1) According to the input feature graph F′ and the word vector set W = {w1, w2, .., w N}, each word vector Where D and N represent the dimension and number of word vectors respectively. W is processed through the fully connected layer to obtain the aggregation matrix W agg , W agg Transpose to get

[0044] (2) Input F′ into the conditional channel attention module and compare it with the maximum pooling result of the input feature F. And average pooling results Perform matrix multiplication to obtain two feature aggregation matrices and

[0045] (3) Assign a weight coefficient to each channel Generate two channel attention maps and Weight coefficient

[0046] is determined by the i-th channel of F and the j-th word in W, and is added element by element. and Merge and activate through sigmoid function σ to get the final conditional channel attention map M c , the calculation formula is:

[0047]

[0048] (4) M c Multiply it with the original input feature F′ to get the new feature map F CC , as the input of the spatial attention module, for F CC Perform channel compression and average pooling respectively and max pooling Operation, get two compressed feature maps, splice the two compressed feature maps in the channel dimension, and perform convolution operation through a 3×3 convolution layer to reduce the feature dimension to 1, and then process it through a sigmoid function to get the spatial attention map M s Finally, M s With F CC Multiply them together to get the final image feature F after spatial attention adjustment SP , the calculation formula is:

[0049]

[0050] F SP =M s ×F CC

[0051] Among them, f 3 × 3 Represents a convolution kernel of size 3×3.

[0052] Beneficial effects:

[0053] The present invention provides a text-to-image synthesis method based on an efficient attention-generated adversarial network, which has the following technical effects:

[0054] 1. The present invention achieves efficient image synthesis by adopting a single-stage generative adversarial network model, and combines it with an efficient attention mechanism to optimize the layout structure of the synthesized image. This mechanism can dynamically adjust the weights of different areas of the image, thereby reasonably arranging the primary and secondary relationships of visual features, and achieving significant enhancement of image synthesis in terms of diversity and semantic consistency.

[0055] 2. The present invention achieves fine adjustment and refinement of feature maps through fusion, iteration and convolution of feature maps, thereby generating a final high-resolution and realistic image, and getting rid of the influence of the initial image quality on the quality of the synthetic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart in an embodiment of the present invention;

[0057] Figure 2 is a schematic diagram of a structure for inputting F into VLIM in an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the structure of inputting F′ into TEM in an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of the structure of a single-stage generative adversarial network in an embodiment of the present invention;

[0060] Figure 5 This is a visual comparison of TV-GAN in an embodiment of the present invention and the most advanced model. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] like Figure 1 As shown, the present invention provides a text-to-image synthesis method based on an efficient attention-generated adversarial network, which is implemented by the following process:

[0063] Step 1: Preprocess the given text and convert each word into a vector of fixed size;

[0064] Step 2: Input the fixed-size word vector in step 1 into the Bi-LSTM, extract the word vector W and sentence vector S from the Bi-LSTM output, and input them into the generator;

[0065] Step 3: Input W and S generated by Bi-LSTM into the generator, and input the noise vector z sampled from the Gaussian distribution, and map the noise vector z to the high-dimensional feature space through the fully connected layer;

[0066] Step 4: Set the upsampling ratio to 2 and use a bilinear interpolation upsampling layer to convert the latent space vector into a 4×4 feature map F.

[0067] Step 5: Input the feature map F in step 4 into the diversity injection module to enhance the diversity of image features;

[0068] Step 6: Iterate steps 4 and 5 three times to obtain a feature map F′ that is gradually enlarged from 4×4 to 32×32;

[0069] Step 7: Input the feature map F′ into the text enhancement module to enhance the semantic consistency of the text;

[0070] Step 8: Fuse the feature maps output by step 7 and step 6, and then iterate the fused feature map 3 times to obtain a feature map F″ with a size of 256×256;

[0071] Step 9: Perform a convolution operation on the feature map F″ obtained in step 8 to generate the final composite image.

[0072] The specific embodiments are as follows:

[0073] Step 1: Preprocess the given text, split it into individual words, build a vocabulary, ensure that each unique word is mapped to a unique index, and convert each word into a fixed-size vector by looking up the pre-trained word vector matrix.

[0074] Step 2: Input the fixed-size word vector into the Bi-LSTM. The forward LSTM processes from the beginning to the end of the sequence, while the reverse LSTM processes from the end to the beginning of the sequence. At each time step, the hidden states of the forward and reverse LSTMs are merged to generate a word vector W containing forward and backward information. Finally, after the Bi-LSTM processes the entire sequence, the hidden state of the last time step is taken as the sentence vector S, which contains the semantic information of the entire sequence and serves as the input of the generator network.

[0075] Step 3: At the same time, input the noise vector z sampled from the Gaussian distribution and map it to the high-dimensional feature space through the fully connected layer. This mapping process is first implemented through a linear transformation, which is defined by the weight matrix w and the bias vector b, where the dimension of the weight matrix w is m×n, m represents the output dimension of the fully connected layer, and n is the dimension of the input noise vector z. The mathematical expression of the linear transformation is Z=w×z+b, where Z is the intermediate result obtained after the linear transformation. Then, the intermediate result Z is processed by the nonlinear activation function f to obtain the output Y=f(Z), thereby introducing nonlinear factors and enhancing the expression ability of the network.

[0076] Step 4: Set the upsampling ratio to 2, and use the bilinear interpolation upsampling layer to convert the latent space vector into a 4×4 feature map F. For each pixel Fij in the 4×4 feature map, determine its corresponding position in the 2×2 feature map, and calculate its distance to the nearest four pixels to determine the weight. Specifically, the coordinates (i, j) of the 4×4 feature map are normalized to the coordinate range of the 2×2 feature map, that is, the normalized row coordinates and column coordinates are divided by 2 respectively to obtain the normalized coordinates (r, c), which are calculated as follows: Normalized row coordinates: r = i / 2; Normalized column coordinates: c = j / 2; where i and j are the row and column indices of the pixel in the 4×4 feature map (i, j∈[0, 3]), respectively. By taking the integer part of the normalized coordinates, that is, using the floor function, the position of the main pixel in the 2×2 feature map (r int , C int ), the calculation formula is: The row index of the main pixel: r int =floor(r); column index of the primary pixel: C int =floor(c). Next, we will calculate the normalized coordinates (r, c) and the rounded coordinates (r int , C int ), which represents the distance between the pixel point Fij and the nearest four pixels in the 2×2 feature map. The calculation formula is: Horizontal distance: dx = rr int ; Vertical distance: dy = cC int . Afterwards, these distances are used to calculate weights, and the calculation formula is: weight v00 = (1-dx)*(1-dy); weight v01 = dx*(1-dy); weight v10 = (1-dx)*dy; weight v11 = dx*dy. Then, the values ​​of the four nearest pixels are weighted averaged according to these weights to obtain the new pixel value. Taking F00 as an example, its pixel value is calculated as follows: F00 = weight v00*v00 + weight v01*v01 + weight v10*v10 + weight v11*v11. This process is repeated until all 16 pixels in the 4×4 feature map are calculated. The specific steps are: starting from F00, calculate the value of each pixel in turn until F33, and we get the complete 4×4 feature map F.

[0077] Step 5: If Figure 2 , the feature map F is input into the diversity injection module (VLIM) to enhance the diversity of image features. First, the feature map F obtained in step 4 and the sentence vector S obtained in step 2 are input into the feature fusion layer of the diversity injection module for fusion processing. The calculation formula is: F′ c =γ c (S)×Fc +β c (S), where γ c (S) and β c (S) are the scale parameter and bias parameter based on the sentence vector S, respectively, and c represents the feature map F c Each channel in (c = 1, 2, ... C). Then, in the processed feature map F' c The noise N is injected into the input, and the calculation formula of the injection operation is: N =F′ c +σN, where σ is a learning variable that is set to 0 at the initial stage of the training process; F N represents the image features after noise processing. Then, the feature map F injected with noise N is obtained. N A 3×3 convolutional layer is input to capture the texture and structure information of the local area. Next, the image features and sentence vector S are further processed by the feature fusion layer, and the output image features are fed into the self-attention mechanism to capture the long-range dependencies in the image, so that the generated local features of the image are coordinated with the global context. In the self-attention mechanism, for each position in the feature map, its query vector is dot-producted with all key vectors to obtain an attention weight matrix SA, which is calculated as: Where Q is the set of all query vectors, K is the set of all key vectors, and d k is the dimension of the key vector, and the softmax function is used to normalize the weight. Then, the calculated attention weight SA is used to perform weighted summation on the value (Value) to obtain the weighted feature representation: Attention(Q, K, V) = AV. Finally, the weighted summation result is concatenated with the original feature map F to obtain the new feature map F SA .

[0078] Step 6: Iterate steps 4 and 5 three times. In each iteration, the size of the feature map is enlarged to twice the original size through upsampling. After these three iterations, the initial feature map is gradually enlarged from 4×4 to 32×32, and finally a high-resolution feature map F′ is obtained.

[0079] Step 7: Figure 3 As shown, F′ is input into the text enhancement module (TEM) to enhance the semantic consistency of the text. According to the input feature graph F′ and the word vector set W = {w1, w2, .., w N}, each word vector Where D and N represent the dimension and number of word vectors respectively. First, W is processed through a fully connected layer to obtain the aggregation matrix W agg , W agg Transpose to get Then, F′ is input into the conditional channel attention module and compared with the maximum pooling result of the input feature F. And average pooling results Perform matrix multiplication to obtain two feature aggregation matrices and Next, assign weight coefficients to each channel to generate two channel attention maps and These weight coefficients is determined by the i-th channel of F and the J-th word in W. By element-by-element addition, and Merge and activate through sigmoid function σ to get the final conditional channel attention map M c The calculation formula is:

[0080]

[0081] Next, M c Multiply it with the original input feature F′ to get the new feature map F CC , as the input of the spatial attention module. In order to extract spatial information, F CC Perform channel compression and average pooling respectively and max pooling The two feature maps are concatenated in the channel dimension and convolved through a 3×3 convolutional layer to reduce the feature dimension to 1. Then, they are processed by the sigmoid function to obtain the spatial attention map M. s Finally, M s With F CC Multiply them together to get the final image feature F after spatial attention adjustment SP , the calculation formula is:

[0082]

[0083] F SP =M s ×F CC

[0084] Among them, f 3×3 Represents a convolution kernel of size 3×3.

[0085] Step 8: Fuse the feature maps output by step 7 and step 6, and then iterate the fused feature map for 3 times. After these 3 iterations, the size of the feature map is gradually enlarged from 32×32 to 256×256, and finally a high-resolution feature map F″ is obtained.

[0086] Step 9: Perform a convolution operation on the iterated 256×256 feature map F″ to fine-tune and refine the feature map, thereby generating the final high-resolution and realistic image.

[0087] like Figure 2 As shown, in this embodiment, we use matching-aware gradient penalty (MA-GP) and one-way output (One-Way Output) technology to optimize our network structure. The calculation formula of the loss function used in the optimization process is: Where (z) represents the image sample generated by the generator G by extracting the latent vector z from the latent space and mapping it to the image space, is the distribution of generated images, and D(·,S) is the discriminator network, which receives the image and sentence vector S as input and outputs a matching score.

[0088]

[0089] Among them, h1(t)=min(0,-1+t) and h2(t)=min(0,-1-t) are hinge losses, which are used to measure the matching degree between real samples and generated samples respectively. represents the real sample, Represents image samples of mismatched data, and k and p are two parameters that balance the gradient. Hinge loss limits the loss range of the discriminator to between -1 and 1, and reduces the instability of the gradient by imposing a gradient penalty on the real data points and their neighboring areas. This measure smoothes the loss function surface around the real data points, which helps the synthetic data points to converge better to the real data points. MA-GP introduces a gradient penalty term in the discriminator's loss function to ensure that the discriminator can stably converge to the real data distribution when processing matching data, thereby avoiding the problem of gradient vanishing or exploding during the training process of the generator. On the other hand, One-Way Output directly calculates the loss by connecting the image features in the discriminator with the sentence vector S and processing it through two convolutional layers. This method enables the gradient to effectively point to the real and matching data points when updating the generator parameters, thereby improving the learning efficiency and convergence speed of the generator.

[0090] The following is a verification experiment of this embodiment:

[0091] The method is compared on two public datasets: CUB and MS-COCO. The statistics of these datasets are shown in Table 1:

[0092] Table 1 Data representation

[0093]

[0094] The implementation details are as follows:

[0095] Our experimental configuration uses Ubuntu 20.04 as the operating system, Pytorch1.11.0 as the deep learning framework, and Cuda 11.3 to accelerate GPU computing. In terms of hardware, we choose NVIDIA RTX 4090 graphics card (with 24GB video memory) as computing resources. During model training, our text encoder uses a bidirectional long short-term memory network to capture the sequence dependencies in text data. For the image generation task, we set the output image resolution to 256×256 pixels to ensure that the generated image has sufficient clarity. We use the Adam optimizer to adjust the network parameters, set β1=0.0001 and β2=0.9, set the learning rate of the generator to 0.0001 and the learning rate of the discriminator to 0.0004. In order to evaluate the visual quality, diversity and semantic consistency of the images generated by TV-GAN, we use the IS and FID indicators, test them on the CUB and MS-COCO datasets respectively, and compare them with a series of advanced text-to-image synthesis methods. The specific data are shown in Table 1.

[0096] Table 2 Comparison results of IS and FID on CUB and COCO test sets

[0097]

[0098] Note: The bold font is the optimal value for each row.

[0099] The numerical comparison results are as follows:

[0100] The comparative experimental results on the CUB dataset show that compared with the AttnGAN model with a stacked structure, our TV-GAN model improves the IS index from 4.28 to 4.93 and reduces the FID index from 24.17 to 15.81. In addition, compared with the DM-GAN, DF-GAN and DE-GAN models with a single-stage structure, TV-GAN achieves the best performance in both IS and FID. On the MS-COCO dataset, the TV-GAN model also achieves the best FID performance, reducing it from 34.53 to 23.31. However, in terms of the IS index, TV-GAN performs slightly worse than other methods. This phenomenon can be attributed to an inherent limitation of the IS index. The Inception model used to calculate the IS is pre-trained on the ImageNet dataset, and the images of ImageNet usually contain only one main object, which is inconsistent with the situation that the images in the MS-COCO dataset usually contain multiple objects, resulting in a deviation in the IS evaluation results. Despite this limitation, the TV-GAN model still demonstrates superior performance in most cases, indicating that our model has a significant advantage in generating diverse and semantically consistent images.

[0101] See also Figure 5 ,The synthesis effects of AttnGAN, DM-GAN, DF-GAN, DE-GAN and our proposed TV-GAN were compared on CUB and MS-COCO datasets. By analyzing the visualization results, we found some key differences. Specifically, the synthetic images of the previous models have various defects, such as Figure 5 The first column shows the missing body parts; the second column shows the distorted body shape; the third column shows the messy feather texture; and the fourth column shows the uncoordinated body proportions. In contrast, the TV-GAN we proposed shows significant advantages in image synthesis. In the generation of bird images, TV-GAN not only ensures the integrity and coordination of the body, but also generates more vivid and realistic complex feather textures, and the posture is more natural. In addition, when generating images of complex scenes on the MS-COCO dataset, despite the shortcomings in some details, TV-GAN still shows better semantic consistency when synthesizing complex scenes. For example, Figure 5 The images in the 6th and 8th columns show that TV-GAN is able to more accurately synthesize the text descriptions of “a cup of milkshake” and “train tracks”.

[0102] The parameter configuration is as follows:

[0103] In the parameter configuration of the TV-GAN model, we set the noise vector z dimension to 100 to provide sufficient diversity so that the generator can capture the potential characteristics and changes of the data; the conditional channel vector dimension is set to 256 to enhance the model's ability to generate high-quality images under specific conditions; the seed number of the random number generator is set to 100 to ensure experimental repeatability; batch_size is set to 32 to balance computing resources and training efficiency; for different datasets, the maximum epoch of the CUB dataset is set to 600 rounds to fully learn complex distributions, while the maximum epoch of the MS-COCO dataset is set to 150 rounds to avoid overfitting and improve efficiency.

[0104] like Figure 4 As shown in the figure, through the above verification experiment, a single-stage generative adversarial network model is used to achieve efficient image synthesis, and an efficient attention mechanism is combined to optimize the layout structure of the synthesized image. This mechanism dynamically adjusts the weights of different regions of the image, thereby reasonably arranging the primary and secondary relationships of visual features, significantly enhancing the performance of image synthesis in terms of diversity and semantic consistency. In this verification experiment, we tested the method on two widely used benchmark datasets and compared the numerical results and visual effects of the method with the existing advanced text-to-image synthesis methods. The results show that the method performs outstandingly in improving the visual quality and diversity of image synthesis, and ensures a high degree of semantic consistency.

[0105] This specific embodiment is only an explanation of the present invention, and it is not a limitation of the present invention. After reading this specification, those skilled in the art can make modifications to this embodiment without creative contribution as needed, but they are protected by the patent law as long as they are within the scope of the claims of the present invention.

[0106] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0107] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text-to-image synthesis method based on an efficient attention-generated adversarial network, characterized in that: The following steps are involved: Step 1: Preprocess the given text and convert each word into a vector of fixed size; Step 2: Input the fixed-size word vector in step 1 into the Bi-LSTM, extract the word vector W and sentence vector S from the Bi-LSTM output, and input them into the generator; Step 3: While inputting the word vector W and sentence vector S into the generator, input the noise vector z sampled from the Gaussian distribution and map the noise vector z to the high-dimensional feature space through the fully connected layer; Step 4: Set the upsampling ratio to 2 and use a bilinear interpolation upsampling layer to convert the latent space vector into a 4×4 feature map F. Step 5: Input the feature map F in step 4 into the diversity injection module to enhance the diversity of image features; Step 6: Iterate steps 4 and 5 three times to obtain a feature map F′ that is gradually enlarged from 4×4 to 32×32; Step 7: Input the feature map F′ into the text enhancement module to enhance the semantic consistency of the text; Step 8: Fuse the feature maps output by step 7 and step 6, and iterate the fused feature maps for 3 times to obtain a feature map F″ with a size of 256×256; Step 9: Perform a convolution operation on the feature map F″ obtained in step 8 to generate the final composite image.

2. According to claim 1, a text-to-image synthesis method based on an efficient attention-generated adversarial network is characterized in that: In step 1, the given text is segmented into individual words and the vocabulary is constructed so that each unique word is mapped to a unique index and each word is converted into a fixed-size vector by looking up the pre-trained word embedding matrix.

3. The text-to-image synthesis method based on efficient attention generative adversarial network according to claim 1 is characterized in that: In step 2, the forward LSTM processes from the beginning to the end of the sequence, and the reverse LSTM processes from the end to the beginning of the sequence. At each time step, the hidden states of the forward and reverse LSTMs are merged to generate a word vector W containing forward and backward information. After the Bi-LSTM processes the entire sequence, the hidden state of the last time step is taken as the sentence vector S as the input of the generator.

4. The text-to-image synthesis method based on efficient attention generative adversarial network according to claim 1, characterized in that: In step three, the process of mapping the noise vector z to the high-dimensional feature space through the fully connected layer includes the following steps: (1) Perform a linear transformation on the noise vector z. The transformation is defined by a weight matrix w and a bias vector b. The dimension of the weight matrix w is m×n, where m represents the output dimension of the fully connected layer and n represents the dimension of the input noise vector z. The mathematical expression of the linear transformation is: Z=W×z+b Where Z is the intermediate result obtained after linear transformation; (2) The intermediate result Z is processed by the nonlinear activation function f to obtain the output: Y = f(Z).

5. The text-to-image synthesis method based on efficient attention generative adversarial network according to claim 1, characterized in that: In step 4, the use of a bilinear interpolation upsampling layer to convert the latent space vector into a feature map F of size 4×4 includes the following steps: (1) Determine the corresponding position of each pixel in the 4×4 feature map F in the 2×2 feature map, calculate the distance between the pixel and the nearest four pixels, and determine the weight; (2) taking a weighted average of the values ​​of the four nearest pixels according to the weights in step (1) to obtain a new pixel value; (3) Repeat steps (1) and (2) until all 16 pixels in the 4×4 feature map are calculated.

6. The text-to-image synthesis method based on efficient attention-generated adversarial network according to claim 1, characterized in that: In step 5, the step of inputting the feature map F into the diversity injection module comprises the following steps: (1) First, the feature map F obtained in step 4 and the sentence vector S obtained in step 2 are input into the feature fusion layer of the diversity injection module for fusion processing. The calculation formula is: F′ c =c c (S)×F c +b c (S) In the formula, γ(S) and β c (S) are the scale parameter and bias parameter based on the sentence vector S, respectively, and c represents each channel in the feature map F; (2) After processing, the feature map F′ c The noise N is injected into the , and the calculation formula of the injection operation is: F N =F′ c +σN Among them, σ is a learning variable, which is set to 0 at the initial stage of the training process; F N Represents the image features after noise processing; (3) Input the image features obtained in step (2) into a 3×3 convolutional layer to capture the texture and structure information of the local area; (4) The image features and sentence vector S are further processed through the feature fusion layer, and the output image features are sent to the self-attention mechanism to capture the long-range dependencies in the image and coordinate the generated local image features with the global context. In the self-attention mechanism, for each position in the feature map, its query vector is dot-producted with all key vectors to obtain an attention weight matrix SA, which is calculated as follows: Where Q is the set of all query vectors, K is the set of all key vectors, and d k is the dimension of the key vector, and the softmax function is used to normalize the weights; (5) Concatenate the weighted summation result with the original feature map F to obtain a new feature map F SA .

7. The text-to-image synthesis method based on efficient attention-generated adversarial network according to claim 1, characterized in that: In step six, in each iteration, the size of the feature map is enlarged to twice its original size through an upsampling operation.

8. The text-to-image synthesis method based on efficient attention-generated adversarial network according to claim 1, characterized in that: In step 7, the feature map F′ is input into the text enhancement module, and the enhancement of text semantic consistency specifically includes the following steps: (1) According to the input feature graph F′ and the word vector set W = {w1, w2, ..., w N }, each word vector Where D and N represent the dimension and number of word vectors respectively. W is processed through the fully connected layer to obtain the aggregation matrix W agg , W agg Transpose to get (2) Input F′ into the conditional channel attention module and compare it with the maximum pooling result of the input feature F. And average pooling results Perform matrix multiplication to obtain two feature aggregation matrices and (3) Assign a weight coefficient to each channel Generate two channel attention maps and Weight coefficient is determined by the ith channel of F and the ith word in W. By element-by-element addition, Hehe Merge and activate through sigmoid function σ to get the final conditional channel attention map M c , the calculation formula is: (4) M c Multiply it with the original input feature F′ to get the new feature map F CC , as the input of the spatial attention module, FCc is channel compressed and average pooled respectively and max pooling Operation, two compressed feature maps are obtained, the two compressed feature maps are spliced ​​in the channel dimension, and convolution operation is performed through a 3×3 convolution layer to reduce the feature dimension to 1, and then processed by a sigmoid function to obtain a spatial attention map M s Finally, M s With F cc Multiply them together to get the final image feature F after spatial attention adjustment sp , the calculation formula is: F sp =M s ×F cc Among them, f 3×3 Represents a convolution kernel of size 3×3.

9. The text-to-image synthesis method based on efficient attention-generated adversarial network according to claim 5, characterized in that: The specific process of determining the weight in step (1) of step 4 is as follows: Normalize the coordinates (i, j) of the 4×4 feature map to the coordinate range of the 2×2 feature map, that is, divide the normalized row coordinates and column coordinates by 2 respectively, and obtain the normalized coordinates (r, c), which are calculated as follows: Normalized row coordinates: r = i / 2; Normalized column coordinates: c = j / 2; where i and j are the row and column indices of the pixel in the 4×4 feature map (i, j∈[0,3]), respectively; by taking the integer part of the normalized coordinates, that is, using the floor function to round down, find the main pixel position (r) in the 2×2 feature map. int , c int ), the calculation formula is: The row index of the main pixel: r int =floor(r); column index of the primary pixel: c int =floor(c); Next, we will calculate the normalized coordinates (r, c) and the rounded coordinates (r int , c int ), which represents the distance between the pixel point Fij and the nearest four pixels in the 2×2 feature map. The calculation formula is: Horizontal distance: dx = rr int ; Vertical distance: dy = cc int ; Afterwards, these distances are used to calculate weights, and the calculation formula is: weight v00 = (1-dx)*(1-dy); weight v01 = dx*(1-dy); weight v10 = (1-dx)*dy; weight v11 = dx*dy.

Images