Text-to-image generation method and system based on progressive multi-granularity semantic information fusion

Abstract

The application provides a text-to-image generation method and system based on progressive multi-granularity semantic information fusion. The generator fuses different granularity semantic information of the text description, can generate an image more consistent with the text description, and solves the problem of inconsistent semantics of the existing single-stage model generated image. For the generator, a text-image consistency loss is added to supervise the generator to generate an image consistent with the text description, thereby enhancing the reality and semantic consistency of the image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, and particularly relates to a text-to-image generation method and system for progressive multi-granular semantic information fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Text-to-image generation is the process of generating realistic and semantically consistent images based on given text descriptions. This research is a promising and important task with broad applications, including art generation, image editing, virtual reality, video games, and computer-aided design. Due to its wide range of applications and challenges, text-to-image generation has become an active field in computer vision and natural language processing.

[0004] Most existing text-to-image generation methods are based on Generative Adversarial Networks (GANs), which can be broadly classified into two categories: multi-stage models and single-stage models.

[0005] The multi-stage model consists of multiple generator / discriminator pairs, each processing feature maps at different scales and refining low-resolution images to obtain high-resolution and visually realistic images. Specifically, the initial network generates a low-resolution rough sketch, which is fed into the next stage network as input. The next stage network outputs a high-resolution image with rich details. Although this text-to-image approach is widely recognized and used and has proven sufficient for generation tasks, it still has several limitations. (1) Entanglement between multiple independent generator / discriminator pairs can lead to generated images that resemble a simple combination of blurry shapes and some details. (2) In the process of generating low-resolution to high-resolution images, the global sentence vector is only used for the generation of the initial image, while the fine-grained text vector is used for the refinement of subsequent images. Each stage network only integrates single-granularity text information, ignoring the constraints of other granular text information, which reduces the consistency between text and image. (3) Training multiple networks simultaneously is not only time-consuming but also detrimental to model convergence. (4) The quality of the final image largely depends on the quality of the initial image.

[0006] Unlike multi-stage models, single-stage models use only one generator / discriminator to generate realistic images that match the given text description. The text-image fusion process is integrated into a single generator, resulting in a simple and easily convergent model that efficiently completes image generation tasks. Although single-stage models demonstrate excellent performance, two problems remain. (1) These models typically use only coarse-grained text information at the sentence level as input, ignoring fine-grained text information at the word level, thus hindering the generation of high-quality images. This problem becomes more severe when generating images with complex scenes. (2) These models only use fully connected layers to connect feature maps and sentence vectors in the global region, lacking precise fusion methods for local regions, which affects the resolution of the final generated image. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a text-to-image generation method and system based on progressive multi-granularity semantic information fusion. By utilizing a generator to fuse semantic information of different granularities from the text description, it can generate images that better match the text description, solving the semantic inconsistency problem present in existing single-stage models. A text-generated image consistency loss is added to supervise the generator in producing images that conform to the text description, enhancing the realism and semantic consistency of the images.

[0008] To achieve the above objectives, a first aspect of the present invention provides a text-to-image generation method for progressive multi-granularity semantic information fusion, comprising:

[0009] Obtain the text description and input the text description into the text encoder to obtain semantic information of different granularities;

[0010] The generator of the constructed adversarial network fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description.

[0011] The generated image, text description, and real image are distinguished by the discriminator of the constructed adversarial network, and the text-generated image consistency loss is calculated. The generator of the adversarial network is trained based on the text-generated image consistency loss.

[0012] A generator from a pre-trained adversarial network is used to generate images that correspond to the text description to be processed.

[0013] A second aspect of the present invention provides a text-to-image generation system for progressive multi-granular semantic information fusion, comprising:

[0014] Acquisition Unit: Acquires text description and inputs the text description into a text encoder to obtain semantic information of different granularities;

[0015] Training Unit: The generator of the constructed adversarial network fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description; the discriminator of the constructed adversarial network distinguishes the generated image, text description and real image and calculates the text-generated image consistency loss; the generator of the adversarial network is trained based on the text-generated image consistency loss.

[0016] Image generation unit: Uses a generator from a trained adversarial network to generate an image corresponding to the text description to be processed.

[0017] A third aspect of the present invention provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a text-to-image generation method of progressive multi-granularity semantic information fusion is performed.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a text-to-image generation method involving progressive multi-granularity semantic information fusion.

[0019] The above one or more technical solutions have the following beneficial effects:

[0020] In this invention, a generator-discriminator single-stage model is adopted, which can skip the process of generating low-resolution images and directly generate high-resolution images. This avoids the entanglement between multiple generator / discriminator pairs in multi-stage models, and is more conducive to model convergence, reducing the time spent training the model.

[0021] In this invention, by utilizing a generator to fuse semantic information of different granularities from the text description, images that better conform to the text description can be generated, thus solving the semantic inconsistency problem existing in the image generation of existing single-stage models. For the generator, a text-generated image consistency loss is added to supervise the generator in generating images that conform to the text description, thereby enhancing the realism and semantic consistency of the images.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a schematic diagram of the adversarial network structure constructed in Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of the adaptive sentence attention fusion module in Embodiment 1 of the present invention;

[0026] Figure 3 This is a schematic diagram of the adaptive word attention fusion module in Embodiment 1 of the present invention;

[0027] Figure 4 This is a schematic diagram of the DownBlock module in Embodiment 1 of the present invention. Figure 5 This diagram illustrates the comparison results of AttnGAN, DFGAN, and the model in this embodiment on the Bird dataset. Figure 6 This diagram illustrates the comparison results of AttnGAN, DFGAN, and the model in this embodiment on the COCO dataset. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0030] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0031] Example 1

[0032] like Figure 1 As shown, this embodiment discloses a text-to-image generation method based on progressive multi-granularity semantic information fusion, including:

[0033] Obtain the text description and input the text description into the text encoder to obtain semantic information of different granularities;

[0034] The generator of the constructed adversarial network fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description.

[0035] The discriminator of the constructed adversarial network is used to identify the consistency between the generated image, text description and real image, and the text-generated image consistency loss is calculated; the generator of the adversarial network is trained based on the text-generated image consistency loss.

[0036] A generator from a pre-trained adversarial network is used to generate images that correspond to the text description to be processed.

[0037] In this embodiment, the text encoder employs a bidirectional long short-term memory (Bi-LSTM) network to learn the semantic representation of a given text description. Specifically, within the Bi-LSTM layer, two hidden states are used to represent semantic information at different granularities, where the intermediate state is represented as the word feature w∈R. 256,18 The final hidden state is represented as sentence features s∈R 256 .

[0038] The adversarial network constructed in this embodiment includes a generator and a discriminator. The generator network consists of seven FBlock modules that process feature maps at different scales. Each FBlock module includes a first convolutional layer, a first normalization layer, an adaptive sentence attention fusion module (ASAF), a second convolutional layer, a second normalization layer, and an adaptive word attention fusion module (AWAF) connected in sequence.

[0039] In this design, both the first and second convolutional layers use 3×3 kernels. Both the first and second normalization layers include Adaptive Layer Instance Normalization (AdaILN) and Affine transformation. AdaILN normalizes the layers and channels of the feature map to stabilize GAN training. Affine extracts semantic cues from the global sentence vector and modulates the normalized feature map to integrate the global sentence vector into the generation process. The normalization layers help control the amount of variation in shape and texture based on the input natural language text.

[0040] The generator network takes a noise vector, a global sentence vector, and word vectors as input, where the noise vector is sampled from a Gaussian distribution, n∈R. 100 First, the concatenated global sentence vector and noise vector are input into the generator network. After passing through a fully connected layer, they are reshaped into a 4x4 feature map. Then, they pass sequentially through the first to seventh FBlock modules, finally outputting a high-resolution 256×256 image.

[0041] like Figure 2 As shown, this is the Adaptive Sentence Attention Fusion (ASAF) module proposed in this embodiment. The adaptive sentence attention fusion module has two inputs: the feature map F output by the first normalization layer. s ∈R C×H×W (where C, H, and W represent F respectively) s The channel dimensions (height and width) and the global sentence vector s∈R M (Where M represents the dimension of sentence embedding). F is then processed using two one-dimensional global average pooling operations. sThe features are aggregated into two separate position-aware feature maps along the horizontal and vertical directions respectively: f sx ∈R C×H×1 and f sy ∈R C×1×W In mathematical terms:

[0042] f sx =GAP X (F s (1)

[0043] f sy =GAP Y (F s (2)

[0044] Among them, GAP X GAP represents global average pooling in the horizontal direction. Y This indicates global average pooling in the vertical direction.

[0045] To model the feature maps of these two specific directions (first-level perceptual feature map f) sx First vertical sensing feature map f sy The long-distance dependency between the two feature maps (f) allows them to capture greater local information in two specific directions, facilitating the focus on feature regions relevant to the input text. This embodiment uses two feature maps, f... sx f sy The data is concatenated using a concat layer, then compressed using a 1×1 convolutional layer. Next, AdaLIN and Non-linear layers are used to encode spatial information in the horizontal and vertical directions to obtain the first feature map. Finally, the first feature map is divided into two features with long-range dependencies, each including both channel and spatial information: the first horizontal query F. sxq ∈R C×H And the first vertical direction query F syq ∈E C×W Mathematically, this is:

[0046] F sxq ,F syq =split(L(AdaLIN(G(f sx +f sy (3)

[0047] Among them, f sx +f sy The concatenation operation represents two vectors in specific directions; G represents a convolutional layer; AdaILN is an adaptive layer-instance normalization; L is a non-linear layer; split represents a splitting operation, which divides a vector into two vectors in specific directions.

[0048] For the global sentence vector, similar to a self-attention mechanism, two different 1×1 convolutional layers are used to obtain two vectors: sentence-level key F. sk ∈R C and value F sv ∈R C Defined as:

[0049] F sk =W sk s,F sv =W sv s (4)

[0050] Among them, W k and W v Let represent two mapping matrices, which are implemented by two different 1×1 convolutions.

[0051] To model the semantic correlation between text vectors and feature maps, the first horizontal direction of the two specific directional vectors, query F, is used. sxq And the first vertical direction query F syq Each is compared with the reshaped text vector key F. sk Perform a dot product operation, then pass it through the softmax function to obtain the attention scores in two directions: the first horizontal direction attention score S. sx ∈R H ×H Attention score in the first vertical direction S sy ∈R W×W This is used to represent the relevance between a sentence and a specific directional subregion.

[0052] S sx =Softmax(D(F sxq T ·F sk (5)

[0053] S sy =Softmax(D(F syq T ·F sk (6)

[0054] Among them, F sxq T and F syq T Represents two query vectors F sxq and F syq The transpose of , D(·) denotes the dot product operation.

[0055] The attention score S obtained sx and S sy respectively with F after reshapesv Perform a dot product operation to obtain the attention weight map of the input text vector and feature vector: W sx ∈R C×H×1 and W sy ∈R C×1×W .

[0056] W sx =Softmax(D(F sv ·S sx T (7)

[0057] W sy =Softmax(D(F sv ·S sy T (8)

[0058] Among them, S sx T and S sy T These represent two attention scores S. sx and S sy The transpose of .

[0059] After obtaining the attention weight map (W) sx W sy After that, it is multiplied with the input of the adaptive sentence attention fusion module to obtain the fine-tuned feature map. Since the original feature map, i.e., the input of the adaptive sentence attention fusion module, contains sufficient information, a residual operation is performed to preserve the original information. The new feature map F, after fusing the sentence information, is obtained by summing the original feature map and the fine-tuned feature map. sn ∈R C×H×W .

[0060] F sn =λ s ·F s ⊙W sx ⊙W sy +F s (9)

[0061] Where λ s is an adaptive parameter that allows the network to adaptively adjust the weights of the feature map to obtain a more suitable feature representation. ⊙ represents the element-wise product operation.

[0062] The ASAF module uses an attention mechanism to highlight regions in the feature map that semantically match the input text, fine-tuning the original feature map to integrate textual information into the generation process. The ASAF module is used once in each Fblock module, applying it to feature maps at different scales to enhance semantic consistency.

[0063] like Figure 3 As shown, the adaptive word attention fusion module (AWAF) proposed in this embodiment has two inputs: the feature map F output by the second normalization layer. e ∈R C×H×W and word context vector e = R M×T Where M is the dimension of word embedding and T is the number of words.

[0064] For feature map F e Perform the same operations as the input of the adaptive sentence attention fusion module; specifically, use global average pooling in two specific directions to obtain the second-level perceptual feature map f. ex ∈R C×H×1 Second vertical sensing feature map f ey ∈R C×1×W Then the two feature maps, i.e. f ex f ey The data is concatenated using a concat layer, then compressed using a 1×1 convolutional layer. Next, AdaLIN and Non-linear layers are used to encode spatial information in the horizontal and vertical directions to obtain the second feature map. Finally, the second feature map is divided into two query feature maps with long-range dependencies and location information: the second horizontal query F... exq ∈R C×H Second vertical direction query F eyq ∈R C×W .

[0065] f ex =GAP X (F e (10)

[0066] f ey =GAP Y (F e (11)

[0067] F exq ,F eyq =split(L(AdaLIN(G(f ax +f ey (12)

[0068] Among them, GAP X and GAP Y These represent global average pooling in the horizontal and vertical directions, respectively; f sx +f sy This represents the concatenation operation of two vectors in a specific direction; G represents a convolutional layer; AdaILN is an adaptive layer-instance normalization, and L is a non-linear layer; split represents a segmentation operation.

[0069] For the word context vector e, similar to the sentence vector processing, it is first processed by two different 1×1 convolutional layers to obtain the word-level key F. ek ∈R C×T and value F ev ∈R C×T .

[0070] F ek =W ek e,F ev =W ev e (13)

[0071] Among them, W ek and W ev These are two learnable mapping matrices, implemented by two distinct 1×1 convolutions.

[0072] To model the semantic relevance between each word and the feature map sub-region, the query and the reshaped F ek Perform a dot product operation, then use the softmax function to obtain the similarity score between the direction vector sub-region and each word: S ex ∈R H ×T and S ey ∈E W×RW This is expressed as follows:

[0073] S ex =Softmax(D(F exq T ·F ek (14)

[0074] S ey =Softmax(D(F eyq T ·F ek (15)

[0075] Where D(·) denotes the dot product operation, F sxq T and F syq T This represents two queries F. exq and query F exq The transpose of .

[0076] Then, the similarity score and the reshaped F ev Perform a dot product operation and then use the softmax function to calculate the attention weights. Obtain the attention weight map W. ex ∈R C×H×1 and W ey ∈R C×1×WThe process is as follows:

[0077] W ex =Softmax(D(F ev ·S ex T (16)

[0078] W ey =Softmax(D(F ev ·S ey T (17)

[0079] Among them, S ex T and S ey T These represent two attention scores S. ex and S ey The transpose of .

[0080] After obtaining the attention weight map W ew W ey Next, it is multiplied with the original feature map, i.e., the input of the adaptive word attention fusion module, to obtain a weighted feature map after fusing word context information. Because the original information is also crucial, a residual structure is used to preserve it, adding the weighted feature map and the original feature map (i.e., the input of the adaptive word attention fusion module) together. By doing so, both the original information is preserved and the fusion of word context information is completed. A new feature map F is obtained. en ∈R C ×H×W The process is as follows:

[0081] F en =λ e ·F e ⊙W ex ⊙W ey +F e (18)

[0082] Where, λ e It is an adaptive parameter that allows the network to adaptively adjust the weights while preserving appropriate fine-tuning features. ⊙ represents element-wise product operation.

[0083] The adaptive word attention fusion module uses an attention mechanism to make the network focus on sub-regions related to words and assign them greater weight, thus generating images that better match the text description. Similarly, it is used once in each Fblock module, acting on feature maps at different scales.

[0084] like Figure 4As shown, the discriminator network of the adversarial network in this embodiment consists of six sequentially connected DownBlock modules. Each DownBlock module employs a residual structure. Each layer of a DownBlock module consists of a sequentially connected 4×4 convolutional layer, a first ReLU layer, a 3×3 convolutional layer, and a second ReLU layer. The shortcut consists of a 1×1 convolutional layer and an average pooling layer to preserve the original features and facilitate model convergence. Each DownBlock reduces the size of the input feature map by half. After six layers, a 4×4 feature map is output.

[0085] The discriminator network takes a real image and a generated image as input, and then passes them through a convolutional layer and six DownBlock modules for feature extraction to obtain visual features I∈R. 256×4×4 The global sentence vector is spatially copied to S∈R 256×4×4 and visual features I∈R 256×4×4 After splicing, two layers of convolution are used to obtain the adversarial loss.

[0086] This embodiment proposes a text-image consistency loss to supervise the generator in producing images that better match the text description. Cosine similarity loss is used as the metric and is defined as follows:

[0087] L sc =1-cos(avg(f(I)),s) (19)

[0088] Where I represents the generated image, s represents the global sentence vector, and f(I)∈R c×4×4 This represents the image features extracted by the discriminator from the generated image. `avg(·)` represents the average pooling operation, which performs average pooling on the image features to compress the information to the same dimension as the sentence vector. `cos(·)` represents the cosine similarity operation, which minimizes the cosine similarity between the generated image vector `f(I)` and the input text vector `s`. The more similar `f(I)` is to `s`, the higher the cosine similarity. sc The smaller the value, the higher the semantic consistency between the generated image and the input text.

[0089] The loss of the discriminator network is:

[0090]

[0091] Where e is the given text description, and e' is the mismatched text description. x is the real image, and z is the noise vector. P r P g Let k and p represent the distribution of the real data and the distribution of the generated data, respectively. The variables k and p are hyperparameters of MA-GP, set to 6 and 2 respectively to balance the effectiveness of the gradient penalty.

[0092] In this implementation, the generator loss includes adversarial loss and text-image consistency loss:

[0093]

[0094]

[0095] Among them, L sc Let λ represent the proposed text-image consistency loss. sc This is the weight of the semantic consistency loss, which is set to 0.2 in this embodiment.

[0096] To verify the generation performance of the model in this embodiment, it was compared with existing state-of-the-art text-to-image generation models on two benchmark datasets, and the results are shown in Table 1.

[0097] Here is a brief introduction to the dataset used and two common evaluation metrics in text-to-image generation.

[0098] Datasets: (1) CUB Birds Dataset. It contains 11,788 bird images, including 200 classes, with 10 corresponding text descriptions for each image. 8,855 images (150 classes) are used as the training set, and 2,933 images (50 classes) are used as the test set. (2) COCO Dataset. It consists of a training set of 80,000 images and a test set of 40,000 images. Each image in COCO has 5 corresponding text descriptions. The COCO dataset differs from the CUB Birds Dataset in that it contains more categories and has a more complex scene distribution, making it a more challenging task for T2I generation.

[0099] Evaluation metrics: Two of the most common evaluation metrics were used: Inception Score (IS) and Fréchet Inception Distance (FID). A higher IS indicates that the generated images are sharper and have high diversity across all categories, and each generated image can be clearly identified as a specific category rather than a vague category. A lower FID indicates that the generative model performs better, with higher image sharpness and greater diversity, indicating a more effective model.

[0100] Table 1

[0101]

[0102]

[0103] Quantitative results show that the model in this embodiment is compared with state-of-the-art GAN methods in text-to-image synthesis on the CUB and COCO datasets. As shown in Table 1, it is clear that the model proposed in this embodiment achieves highly comparable performance. On the CUB dataset, which is rich in descriptive detail, the model with high semantic understanding tends to produce better results than the coarse-grained model, depending on finer-grained semantic fusion. As can be seen from the second column, compared with the state-of-the-art method DF-GAN, the model in this embodiment achieves a significant improvement in FID on the CUB dataset (from 14.81 to 11.77). This is because the model in this embodiment progressively fuses sentence-level and word-level semantic information, learns comprehensive text semantics, and generates images that are closer to real-world distributions. On the COCO dataset, which has multiple objects, complex layouts, and simple details, the model in this embodiment also outperforms other benchmark models. Experimental results show that the model in this embodiment can also synthesize images with multiple objects and complex layouts.

[0104] From the qualitative results, Figure 5 The comparison results of AttnGAN, DFGAN, and the model presented in this embodiment on the Bird dataset are shown. The two baseline models exhibit semantic inconsistencies, bizarre shapes, and unrealistic blurriness. In contrast, the GMF-GAN model in this embodiment generates birds with richer details and more suitable color distribution, and demonstrates higher semantic consistency. This is attributed to the two novel attention modules proposed in this embodiment. These modules match the most suitable feature regions based on the input text and fuse them, progressively integrating sentence and word information into the generation process. Furthermore, the text semantic consistency loss in this embodiment makes the generated image more closely match the input text.

[0105] Figure 6 The comparison results of AttnGAN, DFGAN, and the model of this embodiment on the COCO dataset are shown. As can be seen from the figure, the GMF-GAN of this embodiment produces high-quality, visually realistic, and semantically consistent results on the COCO dataset, indicating that the model of this embodiment also has good generation capabilities for images with complex layouts and multiple objects.

[0106] To verify the performance improvement of ASAF, AWAF, and text-image consistency loss (SCL) proposed in this embodiment, comparative experiments were conducted on the Bird dataset, and the results are shown in Table 2:

[0107] Table 2

[0108]

[0109] The new components in this embodiment's model include an Adaptive Sentence Attention Fusion Module (ASFM), an Adaptive Word Attention Fusion Module (AWFM), and a Text-Image Consistency Loss (SCL). This embodiment quantitatively tests the importance of each component by progressively removing the corresponding parts of the model. (1) Full model (2) SCL removed only (3) AWAF removed only (4) ASAF removed only (5) ASAF and AWAF removed. All results are reported in Table 2.

[0110] By comparing Model 1 (the complete model) and Model 2 (with SCL removed), SCL significantly improved the model's IS (from 4.66 to 4.76) and FID (from 13.30 to 11.77). This indicates that using SCL can significantly improve the quality of the generated images. To test AWAF, Model 1 significantly outperformed Model 3 (with AWAF removed) in IS: 4.44 and FID: 16.05, confirming the necessity of AWAF for fusing fine-grained word-level information. The experimental results comparing Model 1 and Model 4 (with ASAF removed) with IS: 4.37 and FID: 15.38 demonstrate the effectiveness of ASAF for fusing coarse-grained sentence-level information. Finally, Experiment 5 (with ASAF and AWAF removed) shows that the two new attention modules can generate more realistic, reliable, and semantically consistent images by progressively fusing sentence-level and word-level information. This verifies that the modules proposed in this embodiment are beneficial for text information fusion and can generate images that conform to the text description.

[0111] The order in which the ASAF and AWAF modules were added in Fblock was adjusted to verify the impact of progressively coarse-to-fine text feature fusion on the results. The test results are shown in Table 3.

[0112] Table 3

[0113] Norm→ASAF→Norm→ASAF 4.57±0.20 14.84 Norm→AWAF→Norm→AWAF 4.62±0.14 14.03 Norm→ASAF→Norm→AWAF 4.76±0.12 11.77

[0114] The order of sentence and word constraints in the network was adjusted, and quantitative and qualitative results were obtained on the CUB dataset. As shown in Table 3, the order of sentences and words was adjusted to Norm→ASAF→Norm→ASAF, Norm→AWAF→Norm→AWAF, and Norm→ASAF→Norm→AWAF, respectively. Quantitatively, the progressive fusion of sentence-level and word-level semantic information improved the IS from 4.57±0.20 to 4.76±0.12 and the FID from 14.84 to 11.77. This indicates that the progressive fusion of semantic information is more effective.

[0115] Example 2

[0116] The purpose of this embodiment is to provide a text-to-image generation system that integrates progressive multi-granularity semantic information fusion, including:

[0117] Acquisition Unit: Acquires text description and inputs the text description into a text encoder to obtain semantic information of different granularities;

[0118] Training Unit: The generator of the constructed adversarial network progressively fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description; the discriminator of the constructed adversarial network distinguishes the generated image, text description and real image and calculates the text-generated image consistency loss; the generator of the adversarial network is trained based on the text-generated image consistency loss.

[0119] Image generation unit: Inputs the text description to be processed into the generator of the trained adversarial network to obtain the corresponding generated image.

[0120] Example 3

[0121] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0122] Example 4

[0123] The purpose of this embodiment is to provide a computer-readable storage medium.

[0124] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0125] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0126] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0127] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text-to-image generation method based on progressive multi-granularity semantic information fusion, characterized in that, include: Obtain the text description and input the text description into the text encoder to obtain semantic information of different granularities; The generator of the constructed adversarial network fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description; the generator of the adversarial network includes multiple identical FBlock modules connected in sequence, and the FBlock module includes a convolutional layer, a normalization layer, an adaptive sentence attention fusion module, and an adaptive word attention fusion module. The adaptive sentence attention fusion module fuses the global sentence vector with the input feature map of the adaptive sentence attention fusion module through an attention mechanism, specifically including: The input to the adaptive sentence attention fusion module is processed through two one-dimensional global average pooling operations to obtain the first horizontal perceptual feature map and the first vertical perceptual feature map. The first horizontal and first vertical perceptual feature maps are concatenated, and spatial information in the horizontal and vertical directions is encoded by adaptive layer instance normalization and affine transformation. The encoded first feature map is divided into a first horizontal query feature map and a first vertical query feature map; The global sentence vector is processed through different convolutional layers to obtain text key vectors and text value vectors respectively; The first horizontal direction query feature map and the first vertical direction query feature map are respectively multiplied by the text key vector, and then normalized exponential function is used to obtain the first horizontal direction attention score and the first vertical direction attention score. The first horizontal attention score and the first vertical attention score are respectively multiplied by the text value vector to obtain the corresponding attention weight map. The obtained attention weight map is multiplied by the input of the adaptive sentence attention fusion module to obtain the output of the adaptive sentence attention fusion module; The adaptive word attention fusion module fuses word vectors with the input feature map of the adaptive word attention fusion module through an attention mechanism, specifically including: The input of the adaptive word attention fusion module is processed through two one-dimensional global average pooling operations to obtain the second horizontal perceptual feature map and the second vertical perceptual feature map. The second horizontal sensing feature map and the second vertical sensing feature map are concatenated, and spatial information in the horizontal and vertical directions is encoded by adaptive layer instance normalization and affine transformation. The encoded second feature map is segmented into a second horizontal query feature map and a second vertical query feature map. Word vectors are processed through different convolutional layers to obtain word-level key vectors and word-level value vectors respectively; The second horizontal query feature map and the second vertical query feature map are respectively multiplied by the word-level key vector, and then normalized exponential function is used to obtain the second horizontal attention score and the second vertical attention score. The attention weight map is obtained by performing a dot product operation between the second horizontal direction attention score and the second vertical direction attention score and the word-level value vector, respectively. The obtained attention weight map is multiplied by the input of the adaptive word attention fusion module to obtain the output of the adaptive word attention fusion module; The generated image, text description, and real image are distinguished by the discriminator of the constructed adversarial network, and the text-generated image consistency loss is calculated; the generator of the adversarial network is trained based on the text-generated image consistency loss. A generator from a pre-trained adversarial network is used to generate images that correspond to the text description to be processed.

2. The text-to-image generation method based on progressive multi-granularity semantic information fusion as described in claim 1, characterized in that, The text encoder employs a bidirectional long short-term memory network to extract global sentence vectors and word vectors of the text description.

3. The text-to-image generation method based on progressive multi-granularity semantic information fusion as described in claim 1, characterized in that, The discriminator of the adversarial network includes multiple DownBlock modules, which are residual structures. The visual features of the generated image are extracted using multiple DownBlock modules, and the visual features are averaged with the global sentence vector. Then, the result of the average pooling operation is subjected to cosine similarity operation to obtain the text-generated image consistency loss.

4. The text-to-image generation method based on progressive multi-granularity semantic information fusion as described in claim 1, characterized in that, The loss function of the generator in the adversarial network is the sum of the adversarial loss and the text-generated image consistency loss.

5. A text-to-image generation system based on progressive multi-granularity semantic information fusion, characterized in that, A text-to-image generation method for implementing progressive multi-granularity semantic information fusion as described in any one of claims 1-4, comprising: Acquisition Unit: Acquires text description and inputs the text description into a text encoder to obtain semantic information of different granularities; Training Unit: The generator of the constructed adversarial network progressively fuses semantic information of different granularities at different scales to obtain the generated image corresponding to the text description; the discriminator of the constructed adversarial network distinguishes the generated image, text description and real image and calculates the text-generated image consistency loss; the generator of the adversarial network is trained based on the text-generated image consistency loss. Image generation unit: Uses a generator from a trained adversarial network to generate an image corresponding to the text description to be processed.

6. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the text-to-image generation method of progressive multi-granularity semantic information fusion as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the text-to-image generation method of progressive multi-granular semantic information fusion as described in any one of claims 1 to 4.

Images