Multi-modal image synthesis method based on attention mechanism and pyramid pooling

Abstract

The application provides a multi-modal image synthesis method based on an attention mechanism and pyramid pooling, comprising the following steps: S1, using a generator of a generative adversarial network to obtain a feature map g n ; S2, using a pyramid pooling module of the generator to generate a synthetic image X according to the feature map g n ; S3, the generative adversarial network further comprises a discriminator, and a matching perception gradient penalty and a hinge loss function are used to complete training of the generator and the discriminator. The application increases information interaction of different channels, generates a feature map containing more details, and introduces pyramid pooling, so that a synthetic image with more real details is further synthesized.

Description

Technical Field

[0001] This invention belongs to the field of deep image forgery, specifically involving a multimodal image synthesis method based on attention mechanism and pyramid pooling. Background Technology

[0002] Multimodal image synthesis involves converting descriptive input statements into vectors as supplementary information, then constraining generative adversarial networks (GANs) to generate the complete image. This interdisciplinary field of computer vision and natural language processing helps uncover the potential relationships between text and images, forming a visual semantic mechanism for computers. Compared to unimodal image synthesis, multimodal image synthesis involves cross-modal data mapping, making the generation process more challenging. However, unlike unimodal methods which lack control and expressive power over the generated image, multimodal image synthesis offers enhanced control and expressive power. Nevertheless, current multimodal image synthesis methods often result in images lacking detail and visual realism. Summary of the Invention

[0003] This invention proposes a multimodal image synthesis method based on attention mechanism and pyramid pooling, which increases the information interaction between different channels, generates feature maps containing more details, and introduces pyramid pooling to further synthesize more realistic synthetic images.

[0004] This invention is achieved through the following technical solution:

[0005] A multimodal image synthesis method based on attention mechanisms and pyramid pooling includes the following steps:

[0006] Step S1: Obtain feature map g using the generator of a generative adversarial network. n The generator consists of n identical upsampling modules and a pyramid pooling module. Each upsampling module includes an upsampling layer, a first affine layer, a first convolutional layer, a channel attention layer, a second affine layer, and a second convolutional layer. The sentence vector is encoded using a text encoder LSTM, and the encoded sentence vector s is input into the first upsampling module.

[0007] For the first upsampling module:

[0008] After the sentence vector s is sampled by the upsampling layer, the sentence sampling vector s′ is obtained. In the first affine layer, the sentence sampling vector s′ is transformed into the channel scaling parameter γ and the shift parameter β of the affine transformation using a multilayer perceptron. The affine transformation is then performed according to the formula AFF(x|s′)=γ·x+β to obtain the affine feature map x.

[0009] Global mean pooling and global max pooling are performed on each channel of the affine feature map x, and the pooling results are summed to obtain the pooled feature map f. The attention weight w corresponding to the i-th channel of the affine feature map x is then obtained based on the pooled feature map f. i According to the formula x 1-ATT =w i ⊙C i Obtain the output x of the channel attention layer 1-ATT , where C i Let represent the i-th channel of the affine feature map x, where i is an integer;

[0010] The output x of the channel attention layer 1-ATT After passing through the second affine layer and the second convolutional layer, the feature map g1 is obtained and input to the second upsampling module. This feature map g1 is then input to the second upsampling module, and so on, until the feature map g output by the nth upsampling module is obtained. n ;

[0011] Step S2: The generator's pyramid pooling module, based on the feature map g... n Generate a synthetic image X: combine feature map g n The feature map g is divided into sub-regions of size m×m, with m taking different values ​​to obtain different scale sub-regions. For each scale sub-region, the feature map g is processed accordingly. n Average pooling is performed, and the number of channels in each pooling output is convolved to reduce dimensionality. The dimensionality-reduced result is then upsampled using bilinear interpolation to obtain the size and feature map g. n The same recovery feature map, these recovery feature maps and feature map g n The pyramid pooling global feature map is obtained by concatenating and adding the channel dimensions, and the synthesized image X is generated based on the pyramid pooling global feature map.

[0012] Step S3: The generative adversarial network also includes a discriminator, which is trained using a matching-aware gradient penalty and a hinge loss function.

[0013] Furthermore, in step S1, n = 6 upsampling modules are set.

[0014] Furthermore, in step S1, the generative adversarial network also has a fully connected layer. Random noise z is used as the input of the fully connected layer. The fully connected layer maps the random noise z into a three-dimensional tensor concatenated with the sentence vector and then inputs it into the first upsampling module.

[0015] Furthermore, in step S1, according to the formula Global mean pooling is performed on each channel of the affine feature map x, according to the formula... Global max pooling is performed on each channel of the affine feature map x, according to the formula wi =σ(conv(f) i Obtain attention weights w i ,in, conv(·) is a 1×1 convolution operation, σ is the sigmoid activation function, and W i H represents the width of the i-th channel. i It represents the height of the i-th channel.

[0016] Furthermore, in step S2, m takes values ​​of 1, 2, 3, and 6 respectively, meaning the feature map g... n It is divided into four sub-regions with sizes of 1×1, 2×2, 3×3 and 6×6.

[0017] Furthermore, in step S2, according to formula X PPM =Cat[BU(Conv(X) C×1×1 )),BU(Conv(X C×2×2 )),BU(Conv(X C×3×3 )),BU(Conv(X C×6×6 )), g6] obtains the pyramid pooling global feature map, where X C×1×1 =AP C×1×1 (g6), X C×2×2 =AP C×2×2 (g6), X C×3×3 =AP C×3×3 (g6), X C×6×6 =AP C×6×6 (g6), AP(·) is the average pooling operation, Conv(·) is the 1×1 convolution operation, BU(·) is the bilinear interpolation upsampling operation, and Cat(·) is the connection function.

[0018] Furthermore, in step S2, the global features of pyramid pooling are processed. Figure X PPM A 1×1 convolution operation is performed to obtain the synthesized image X.

[0019] Furthermore, in step S3, the total loss function L of the discriminator... D for in: Let the hinge loss function of the discriminator be calculated according to the formula. Calculate, where P r Let x′ be the real image distribution matching the text, D(·) represent the discriminator computation, E be the expectation, s be the sentence vector, and P be the real image distribution matching the text. g For the distribution of generated images that match the text, P mis For the true image distribution that does not match the text; L MA-GP To match the loss function of the perceptual gradient penalty, according to the formula Calculate, where k and p are hyperparameters. This is the gradient operator.

[0020] The present invention has the following beneficial effects:

[0021] 1. This invention utilizes a channel attention mechanism after the affine layer and convolutional layer to focus on important channels of the feature map. The attention mechanism is used to weight the feature map, increasing the information interaction between different channels and making the generated image more detailed. On the other hand, pyramid pooling is used to take into account both the contextual information and global information of the feature map, thereby enhancing the details and edges of the feature map, while also enhancing the overall brightness, contrast and other macroscopic features of the generated image, making the details of the generated image more realistic. Attached Figure Description

[0022] The present invention will now be described in further detail with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of the steps of the present invention.

[0024] Figure 2 This is a flowchart illustrating the overall framework of the present invention.

[0025] Figure 3 This is a network structure diagram of the adversarial generator of the present invention.

[0026] Figure 4 This is a diagram of the channel attention structure of the present invention.

[0027] Figure 5 This is a diagram of the pyramid pooling structure of the present invention.

[0028] Figure 6 This is a comparison diagram of bidirectional discriminator networks and unidirectional discriminator networks.

[0029] Figure 7 A schematic diagram illustrating the perceptual gradient penalty.

[0030] Figure 8 Example diagram of the CUB dataset.

[0031] Figure 9 Example graph for the COCO dataset.

[0032] Figure 10 Example image for the Multi-Modal CelebA-HQ dataset.

[0033] Figure 11 The figures show the experimental results of this invention and other existing methods.

[0034] Figure 12 This is a comparison graph showing the qualitative ablation experimental effects of the present invention and other existing methods. Detailed Implementation

[0035] like Figure 1 and Figure 2 As shown, the multimodal image synthesis method based on attention mechanism and pyramid pooling includes the following steps:

[0036] Step S1: Obtain feature map g using the generator of a generative adversarial network. n The generator consists of n=6 identical upsampling modules and a pyramid pooling module, such as... Figure 3 As shown, each upsampling module includes an upsampling layer, a first affine layer, a first convolutional layer, a channel attention layer, a second affine layer, and a second convolutional layer. The text encoder LSTM is used to encode the sentence vector, and the encoded sentence vector s is input into the first upsampling module.

[0037] Generative adversarial networks also have fully connected layers. Random noise z is taken as input to the fully connected layer. The fully connected layer maps the random noise z into a three-dimensional tensor concatenated with the sentence vector, which is then input to the first upsampling module. This process is represented as g0 = F0(z), where F0 is the fully connected layer. The output of the first upsampling module can be represented as g1 = F1. UP (g0,s), the outputs of the other five upsampling modules can be expressed as g j =F j UP (g j-1 ,s),j=2,3,...,6,where,F j UP For upsampling modules;

[0038] For the first upsampling module:

[0039] After the sentence vector s is sampled by the upsampling layer, the sentence sampling vector s′ is obtained. In the first affine layer, the sentence sampling vector s′ is transformed into the channel scaling parameter γ and the shift parameter β of the affine transformation using a multilayer perceptron. The affine transformation is then performed according to the formula AFF(x|s′)=γ·x+β to obtain the affine feature map x.

[0040] Among them, the channel scaling parameter γ is calculated as γ=MLP1(s′), and the displacement parameter β is calculated as β=MLP2(s′). MLP1(·) and MLP2(·) are both multilayer perceptrons.

[0041] According to the formula Global mean pooling is performed on each channel of the affine feature map x, according to the formula... Perform global max pooling on each channel of the affine feature map x, and sum the pooling results to obtain the pooled feature map. And according to formula w i =σ(conv(f)i The attention weight w corresponding to the i-th channel of the affine feature map x is obtained. i After obtaining the attention weights for each channel, the formula x is used. 1-ATT =w i ⊙C i Obtain the output x of the channel attention layer 1-ATT , where C i Let denot be the i-th channel of the affine feature map x, where i is an integer, conv(·) is a 1×1 convolution operation, σ is the sigmoid activation function, and W i H represents the width of the i-th channel. i The channel attention structure diagram is shown below, where ⊙ represents the height of the i-th channel and ⊙ represents the element-wise product. Figure 4 As shown;

[0042] The output x of the channel attention layer 1-ATT After passing through the second affine layer and the second convolutional layer in sequence, the feature map g1 is obtained and input to the second upsampling module. This feature map g1 is then input to the second upsampling module. In the second upsampling module, the feature map g1 is upsampled and then passed through the first affine layer, the first convolutional layer, the channel attention layer, the second affine layer, and the second convolutional layer in sequence to obtain the feature map g2 output by the second upsampling module. This feature map g2 is also used as the input of the third upsampling module, and so on, until the feature map g6 output by the sixth upsampling module is obtained.

[0043] Step S2: The generator's pyramid pooling module, based on the feature map g... n Generate a synthetic image X: combine feature map g n The feature map g is divided into sub-regions of size m×m, with m taking different values ​​to obtain different scale sub-regions. For each scale sub-region, the feature map g is processed accordingly. n Average pooling is performed, and the number of channels in each pooling output is convolved to reduce dimensionality. The dimensionality-reduced result is then upsampled using bilinear interpolation to obtain the size and feature map g. n The same recovery feature map, these recovery feature maps and feature map g n The pyramid pooling global feature map is obtained by concatenating and adding the channel dimensions, and the synthesized image X is generated based on the pyramid pooling global feature map.

[0044] Global and contextual information of feature maps are crucial to the quality of synthesized images. Therefore, pooling operations at different scales are used to capture the contextual information of feature maps, reduce the loss of contextual information between feature maps at different scales, thereby enhancing the details and edges of feature maps. At the same time, global scene prior information is constructed on the final feature map to improve the network's ability to acquire global information.

[0045] Specifically, such as Figure 5As shown, m takes values ​​of 1, 2, 3, and 6, which means the feature map g n The feature map is divided into four sub-regions of different sizes: 1×1, 2×2, 3×3 and 6×6. The 1×1 scale features are the coarsest, while the 6×6 scale features are more refined. In this case, the multi-scale feature map aggregates the overall and detailed information of the feature map, obtains global information and expands the receptive field.

[0046] According to the formula Obtain global features of pyramid pooling Figure X PPM For this pyramid pooling global feature Figure X PPM Performing a 1×1 convolution operation yields the synthesized image X, i.e., X = Conv(X PPM ), where X PPM ∈R 2C×H×W , X∈R C×H×W X C×1×1 =AP C×1×1 (g6), X C×2×2 =AP C×2×2 (g6), X C×3×3 =AP C×3×3 (g6), X C×6×6 =AP C×6×6 (g6), AP(·) is the average pooling operation, Conv(·) is the 1×1 convolution operation, BU(·) is the bilinear interpolation upsampling operation, and Cat(·) is the connection function that connects the feature maps along the vertical channel;

[0047] Step S3, Model Training: The Generative Adversarial Network also includes a discriminator, which determines whether the image is real and whether it matches the text. The generator and discriminator are trained using the matching-aware gradient penalty and the hinge loss function.

[0048] For stacked multimodal image synthesis networks, the discriminator needs to determine two issues: whether the image is realistic and whether it matches the text. Bidirectional discrimination requires calculating unconditional and conditional losses, summing them to obtain the adversarial loss. During the iterative process, the adversarial loss deviates from the first two losses during convergence, potentially leading to mode collapse.

[0049] The discriminator in this embodiment uses matched-aware gradient penalty loss, and the discriminator converges based on one-way discrimination. Figure 6 This section compares bidirectional and unidirectional discriminator networks. The advantage of matched-aware gradient penalty is that it ensures the discriminator has a smooth loss function based on the gradient penalty, using gradient descent to help the generator converge to the desired value. A schematic diagram of matched-aware gradient penalty is shown below. Figure 7 As shown.

[0050] In the discriminator of this embodiment, the adversarial loss is a hinge loss, and the hinge loss of the discriminator is specifically as follows: Generator adversarial loss is The loss function of the generator in a generative adversarial network is Among them, P r Let x′ be the real image distribution for matching with the text (i.e., sentence vectors), D(·) denote the discriminator computation, E be the expectation, X be the synthetic image, s be the sentence vector, and P be the real image distribution for matching with the text (i.e., sentence vectors). g For the distribution of generated images that match the text, P mis For the true image distribution that does not match the text; L MA-GP The loss function for matching-aware gradient penalty in text-to-image generation is based on the formula... The calculation is performed, where k and p are hyperparameters, set to 2 and 6 during training. This is the gradient operator.

[0051] To fully verify the effectiveness of the channel attention mechanism and pyramid pooling introduced in this invention, experiments were conducted on three datasets: the CUB bird dataset, the COCO dataset, and the Multi-Modal CelebA-HQ dataset.

[0052] The CUB bird dataset, built by the California Institute of Technology, contains 11,788 images of 200 bird species. Each bird species has a text description with specific grammar, and each image corresponds to a description in 10 languages, such as... Figure 8 As shown. This invention selects 8855 images of 150 bird species as the training set and 2933 images of 50 bird species as the test set.

[0053] The COCO dataset, built by Microsoft, is a large dataset for various vision-based tasks, such as image recognition, image segmentation, and caption display. The dataset contains features of a wide variety of images, and for each image, five different manually annotated captions, such as... Figure 9 As shown, the dataset contains images of real objects with multiple objects, as well as images of single instances, such as airplanes, people, and animals. This invention selects 80K images as the training set and 40K images as the test set.

[0054] The Multi-Modal CelebA-HQ dataset

[44] is a large-scale face image dataset. Because descriptive text has been added, the Multi-Modal CelebA-HQ dataset is no different in data format from other common datasets, such as the COCO dataset and the CUB dataset; their data formats are identical. Figure 10As shown. This invention selects 24K images as the training set and 6K images as the test set.

[0055] The evaluation metric of this invention is FID (Frame ID), which is a measure of the difference between the generated image and the real image. Specifically, it involves extracting features from both the real and generated images and calculating and comparing their feature distributions. Where, μ r and ε r It is the feature mean and covariance matrix of the real image, μ g and ε g is the feature mean and covariance matrix of the generated image, and Tr(·) is the trace of the matrix.

[0056] The size of FID is inversely proportional to image quality; a smaller FID rating indicates that a more realistic image has been generated.

[0057] This invention selected three representative methods from existing technologies as baseline methods for comparative experiments, as detailed below:

[0058] AttnGAN method: Achieve fine-grained multimodal image synthesis through attention-driven multi-stage refinement. With the help of attention-generating networks, AttnGAN focuses on text words. The model stacks three generator networks and three discriminator networks to complete the generation of low-resolution images into high-resolution images.

[0059] The DM-GAN method utilizes a memory mechanism to determine which information in the text is important, making the generated images more consistent with the text description. This model solves the problems of insufficient detail and lack of consistency with the text description in the generated images to a certain extent, but it also stacks two stages of image generation.

[0060] The DF-GAN method proposes a single-stage multimodal image synthesis framework. It establishes a generator and discriminator through residual connections and introduces affine transformation and gradient penalty to make the generated images more consistent with the text. However, the DF-GAN method still has the problem of insufficient detail synthesis.

[0061] Of the three methods mentioned above, AttnGAN and DM-GAN are based on stacked generative adversarial networks, while DF-GAN is based on a single-stage generative adversarial network.

[0062] Comparative experimental analysis:

[0063] The present invention was experimentally compared with the three baseline methods mentioned above. The experimental results are shown in Table 1. The best results are shown in bold, and the second-best results are shown in underlined. The effect of the synthesized image of the present invention is as follows: Figure 11 As shown.

[0064] Table 1: Comparative Experiment Results (FID)

[0065]

[0066]

[0067] For the CUB dataset, the FID of this invention is the lowest, indicating that the images synthesized by this invention are the most realistic in the synthesis of multimodal bird images, and the fusion of attention mechanism and pyramid pooling is effective. The FID values ​​of the three baseline methods are all low, indicating that the three baseline methods can basically synthesize relatively realistic bird images, but are not as good as this invention in terms of detail synthesis. The FID of this invention is only 3.59 lower than the suboptimal method. Both this invention and the suboptimal method are based on single-stage generative adversarial networks, indicating that single-stage generative adversarial networks are effective in generating bird images, while methods based on stacked generative adversarial networks compromise the realism of the generated bird images to some extent. This invention can not only synthesize bird images that conform to the text description, but also synthesize bird images with rich and realistic details.

[0068] For the COCO dataset, the FID of this invention is the lowest, indicating that in multi-scene, multimodal image synthesis, the multi-scene images synthesized by this invention are closest to real images. The FID of this invention is 6.22 lower than the suboptimal method, showing a significant reduction. This is because multi-scene images are relatively complex. This invention first uses a channel attention mechanism to enhance the information interaction between channels, making the generated scene richer; then, it utilizes pyramid pooling to fuse contextual and global information, enhancing details and edges, and processing overall brightness and contrast, making the generated scene more realistic. The multi-scene images synthesized by this invention are rich in content and have details, but further improvement is needed.

[0069] For the Multi-Modal CelebA-HQ dataset, the present invention exhibits the lowest FID, indicating that in multi-scene face image synthesis, the synthesized face images are closest to real images. The FID of the present invention is 3.44 lower than the suboptimal method, showing a slight decrease; the FID values ​​of the three baseline methods are all relatively high. This is because people in images may wear accessories, have different hairstyles, and possess different facial expressions, making it difficult to capture details when generating complex character images. The present invention can synthesize character images that match the text description, but background artifacts still exist for some characters.

[0070] Ablation experiment analysis:

[0071] To verify the contributions of channel attention mechanism and pyramid pooling to multimodal image synthesis tasks, this invention designed ablation experiments on the CUB and COCO datasets. The experimental results are shown in Table 2.

[0072] Table 2: Results of Quantitative Ablation Experiment (FID)

[0073]

[0074] For the CUB dataset, when channel attention is introduced into the DF-GAN method to adjust channel information weights, the distance (FID) between the generated image and the real image decreases by 3.21. This is because the main object in the image is a bird, and channel attention increases the information interaction between different channels, making the generated image more detailed, such as enriching details of feathers, head, beak, and pose. When pyramid pooling is introduced into the DF-GAN method, the distance (FID) between the generated image and the real image decreases by 2.84. Pyramid pooling obtains contextual and global information, making the details of the generated image more realistic, such as enhancing bird details and edges, while also paying attention to the texture of bird feathers and image brightness. When both channel attention and pyramid pooling are introduced, the distance (FID) between the generated image and the real image decreases by 3.59. The combination of the two further synthesizes more realistic bird images.

[0075] For the COCO dataset, when channel attention is introduced into the DF-GAN method to adjust channel information weights, the distance between the generated image and the real image (FID) decreases by 5.43. The COCO dataset has complex scenes, and channel attention increases information interaction between different channels, focusing on important parts of the synthesized scene structure, making the scene of the synthesized image richer. When pyramid pooling is introduced into the DF-GAN method, the distance between the generated image and the real image (FID) decreases by 5.65. When synthesizing images with complex scenes, the FID decreases more with pyramid pooling than with channel attention, indicating that the contextual and global information obtained by pyramid pooling is more beneficial for synthesizing complex scenes. When both channel attention and pyramid pooling are introduced, the distance between the generated image and the real image (FID) decreases by 6.22. The combination of the two further improves the ability to synthesize images of complex scenes.

[0076] Comparison of qualitative ablation experimental effects, such as Figure 12 As shown, the performance of this invention is demonstrated using visualization results on both the CUB and COCO datasets. Slightly deficient areas are marked with black boxes in the figures. Qualitative ablation experiments were conducted by comparing the images generated by base methods 1, 2, and 3, along with the final images generated by this invention, all under the same textual description conditions.

[0077] For the CUB dataset, without channel attention and pyramid pooling, basic method 1 captures the color features of "yellow bird," "black trail," and "small," as well as the shape feature of "small." However, it lacks in the synthesis of fur texture and claws, and the detailed information "yellow along the top of the feathers" is not clearly synthesized. When channel attention is introduced into the DF-GAN method to adjust the channel information weights, it can be seen that basic method 2 not only captures the main color and shape features but also shows some improvement in the synthesis of detailed fur and claws. However, the detailed information "yellow along the top of the feathers" is still not clearly synthesized. When pyramid pooling is introduced into the DF-GAN method, it can be seen that basic method 3 captures the synthesis of the main objects, synthesizes relatively realistic fur texture, and synthesizes claw details well. However, compared to basic methods 1 and 2, the detailed information "yellow along the top of the feathers" is still not clearly synthesized. The "feathers" synthesis is less obvious; when channel attention mechanism and pyramid pooling are introduced at the same time, the present invention not only captures the main color features and shape features, but also synthesizes images that conform to the text description, the synthesized details are more realistic, and a more realistic background is synthesized.

[0078] For the COCO dataset, without channel attention and pyramid pooling, basic method 1 captures the main text information, and the synthesized image roughly matches the text. However, due to the complexity of the scene, it is difficult to synthesize normal human poses, and the contrast between light and dark under natural light is also difficult to synthesize, resulting in a lack of detail and overall quality in the image. When channel attention is introduced into the DF-GAN method to adjust the channel information weights, more scene elements are synthesized in the image, but the human pose is slightly distorted, artifacts exist in the scene, and the texture of the snow in the synthesized image is not realistic enough. When pyramid pooling is introduced into the DF-GAN method, the human pose in the synthesized image is more normal compared to basic methods 1 and 3. When both channel attention and pyramid pooling are introduced, this invention takes into account both human pose and background synthesis. Compared to the previous three basic methods, this invention synthesizes more elements and has more realistic details, but it still has some shortcomings when synthesizing complex scene images.

[0079] In summary, the experimental results of this invention demonstrate that, compared with mainstream methods, this invention achieves superior results on all three datasets.

[0080] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the specification of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A multimodal image synthesis method based on attention mechanism and pyramid pooling, characterized in that: Includes the following steps: Step S1: First, obtain the feature map g using the generator of the generative adversarial network. n The generator consists of n identical upsampling modules. Each upsampling module includes an upsampling layer, a first affine layer, a first convolutional layer, a channel attention layer, a second affine layer, and a second convolutional layer. The sentence vector is encoded using a text encoder LSTM, and the encoded sentence vector s is input into the first upsampling module. For the first upsampling module: After the sentence vector s is sampled by the upsampling layer, the sentence sampling vector s′ is obtained. In the first affine layer, the sentence sampling vector s′ is transformed into the channel scaling parameter γ and the shift parameter β of the affine transformation using a multilayer perceptron. The affine transformation is then performed according to the formula AFF(x|s′)=γ·x+β to obtain the affine feature map x. Global mean pooling and global max pooling are performed on each channel of the affine feature map x, and the pooling results are summed to obtain the pooled feature map f. The attention weight w corresponding to the i-th channel of the affine feature map x is then obtained based on the pooled feature map f. i According to the formula x 1-ATT =w i ⊙C i Obtain the output x of the channel attention layer 1-ATT , where C i Let represent the i-th channel of the affine feature map x, where i is an integer; The output x of the channel attention layer 1-ATT After passing through the second affine layer and the second convolutional layer, the feature map g1 is obtained and input to the second upsampling module. This feature map g1 is then input to the second upsampling module, and so on, until the feature map g output by the nth upsampling module is obtained. n ; Step S2: The generator generates the feature map g. n Generate a synthetic image X: Combine feature map g n The feature map g is divided into sub-regions of size m×m, with m taking different values ​​to obtain different scale sub-regions. For each scale sub-region, the feature map g is processed accordingly. n Average pooling is performed, and the number of channels in each pooling output is convolved to reduce dimensionality. The dimensionality-reduced result is then upsampled using bilinear interpolation to obtain the size and feature map g. n The same recovery feature map, these recovery feature maps and feature map g n The pyramid pooling global feature map is obtained by concatenating and adding the channel dimensions, and the synthesized image X is generated based on the pyramid pooling global feature map. Step S3: The generative adversarial network also includes a discriminator, which is trained using a matching-aware gradient penalty and a hinge loss function.

2. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 1, characterized in that: In step S1, n = 6 upsampling modules are set.

3. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 2, characterized in that: In step S1, the generative adversarial network also has a fully connected layer. Random noise z is used as the input of the fully connected layer. The fully connected layer maps the random noise z into a three-dimensional tensor concatenated with the sentence vector and then inputs it into the first upsampling module.

4. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 1, 2, or 3, characterized in that: In step S1, according to the formula Global mean pooling is performed on each channel of the affine feature map x, according to the formula... Global max pooling is performed on each channel of the affine feature map x, according to the formula w i =σ(conv(f) i Obtain attention weights w i ,in, conv(·) is a 1×1 convolution operation, σ is the sigmoid activation function, and W i H represents the width of the i-th channel. i It represents the height of the i-th channel.

5. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 1, 2, or 3, characterized in that: In step S2, m takes values ​​of 1, 2, 3, and 6 respectively, which means the feature map g n It is divided into four sub-regions with sizes of 1×1, 2×2, 3×3 and 6×6.

6. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 2 or 3, characterized in that: In step S2, according to formula X PPM =Cat[BU(Conv(X) C×1×1 )),BU(Conv(X C×2×2 )),BU(Conv(X C×3×3 )),BU(Conv(X C×6×6 )), g6] obtains the pyramid pooling global feature map, where X C×1×1 =AP C×1×1 (g6), X C×2×2 =AP C×2×2 (g6), X C×3×3 =AP C×3×3 (g6), X C×6×6 =AP C×6×6 (g6), AP(·) is the average pooling operation, Conv(·) is the 1×1 convolution operation, BU(·) is the bilinear interpolation upsampling operation, and Cat(·) is the connection function.

7. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 6, characterized in that: In step S2, the pyramid pooling global feature map X is processed. PPM A 1×1 convolution operation is performed to obtain the synthesized image X.

8. The multimodal image synthesis method based on attention mechanism and pyramid pooling according to claim 1, 2, or 3, characterized in that: In step S3, the total loss function L of the discriminator is... D for in: Let the hinge loss function of the discriminator be calculated according to the formula. Calculate, where P r Let x′ be the real image distribution matching the text, D(·) represent the discriminator computation, E be the expectation, s be the sentence vector, and P be the real image distribution matching the text. g For the distribution of generated images that match the text, P mis For the true image distribution that does not match the text; L MA-GP To match the loss function of the perceptual gradient penalty, according to the formula Calculate, where k and p are hyperparameters. This is the gradient operator.

Images