Text-to-image generation method and system based on spatial perception decoupling

Abstract

The application discloses a text-to-image generation method and system based on spatial perception decoupling, relates to the technical field of image processing, and comprises the following steps: receiving a training data set, inputting the training data set into a pre-established spatial perception generative adversarial network model for training, obtaining an enhanced text vector from the text through a text encoder, decoupling the vector into layer-specific spatial perception image codes of different scales by a spatial perception image code mapper, inputting the layer-specific spatial perception image codes and the enhanced text vector into each layer of an image generator as conditions to generate a text corresponding image, inputting the text corresponding image into a discriminator for training, taking improving discrimination ability as the target, and outputting a trained spatial perception generative adversarial network model; receiving a text to be input, inputting the text to be input into the trained spatial perception generative adversarial network model, and outputting a corresponding image of the input text. Thus, an image highly matched with the text description is generated.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a text-to-image generation method and system based on spatial awareness decoupling. Background Technology

[0002] With the rapid development of deep learning, generative adversarial networks have made breakthrough progress in the field of image generation. Among them, text-to-image generation is an important application direction of generative adversarial networks, aiming to generate corresponding visual images through text descriptions and realize cross-modal conversion from "language semantics" to "visual content". At present, existing text-to-image generation models have shown good performance in simple scenes (such as flowers and faces), but there are still significant limitations in the generation tasks of complex images (containing multiple objects, diverse shapes and complex spatial relationships): (1) Limited generation capability: Existing models generally directly map low-dimensional 1-D text vectors to high-dimensional 3-D images (channels × height × width), lacking an intermediate decoding process. The text vectors of complex images contain compact and deeply encoded visual semantics (such as background, object category, spatial layout, etc.), while the generator needs to complete the dual tasks of "decoding visual semantics" and "synthesizing visual features" at the same time, which leads to limited generation capability and difficulty in accurately restoring the details and structure of complex scenes. (2) Insufficient text-image fusion capability: Existing models have defects in their text-image fusion modules: some use channel-level fusion, ignoring the modulation of visual features in the spatial dimension; others use pixel-level cross-attention, causing the fusion process to rely on image features generated rather than textual conditions. Neither approach can fully utilize the spatial information in the text, resulting in low semantic consistency between the generated image and the text description, especially in the reconstruction of spatial relationships between multiple objects. Although some studies have attempted to introduce additional spatial conditions (such as layout and semantic masks) to improve generation quality, these methods increase input complexity and training dependence, limiting the model's versatility. Therefore, how to improve the generation capability of complex images and the text-image fusion capability without introducing additional input has become a key problem that urgently needs to be solved in the field of text-to-image generation. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, the present invention aims to provide a text-to-image generation method and system based on spatially aware decoupling.

[0004] Firstly, the objective of this invention can be achieved through the following technical solution: a text-to-image generation method based on spatially aware decoupling, the method comprising the following steps:

[0005] The training dataset is received and input into a pre-established spatially aware generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatially aware generative adversarial network model includes a text encoder, a spatially aware image code mapper, an image generator, and a discriminator.

[0006] The text data is used to generate enhanced text vectors based on a text encoder. The enhanced text vectors are then used to generate layer-specific spatially sensed decoupled image codes based on a pre-built spatially sensed image code mapper. The layer-specific spatially sensed decoupled image codes and the enhanced text vectors are used as conditional inputs to each layer of the image generator to generate text-corresponding images. The text-corresponding images are then input into the discriminator for training. The goal is to improve the discriminative ability of image and text matching. The output is the trained spatially sensed generative adversarial network model.

[0007] The system receives the text to be input, feeds it into the trained spatial awareness generative adversarial network model, and outputs the corresponding image of the input text.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: each image in the training dataset corresponds to one or more corresponding text descriptions, the text descriptions include objects, shapes and spatial relationships in the image, and the training dataset includes simple scenes containing a single object and complex scenes containing multiple objects.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the text encoder employs a pre-trained bidirectional LSTM network, the expression of which is as follows:

[0010]

[0011] Where T is the input text and e is the encoded text vector.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the spatially aware image code mapper comprising multiple image code modulation modules and layer-specific adapter modules; the generator comprising multiple upsampling generation blocks integrating a dual fusion module; the discriminator comprising multiple discriminator blocks and convolutional layers; the image code modulation module performing spatial information modulation on the initial image code based on text; the layer-specific adapter module being used to convert the global image code into layer-specific image codes corresponding to different layers in the image generator; and the dual fusion module performing spatial and channel dimension fusion based on the layer-specific image code and the enhanced text vector, respectively.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the discriminator differentiation process includes:

[0014] Input text and real image, input text and generated image, and input text and non-matching real image;

[0015] The discriminator assigns a high score when input text and a real image are input together; a low score when input text and a generated image are input together; and a low score when input text and a mismatched real image are input together. The score ranges from -1 to 1, inclusive, with -1 being the lowest score and 1 being the highest.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: during training, the discriminator employs hinge loss and gradient penalty to stabilize the training process.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the expression of the pre-established spatially aware generative adversarial network model is as follows:

[0018]

[0019] in, For the loss of the discriminator, As expected, For input image, For the true distribution of image data, To find the minimum value function, For discriminator, For text vectors that match the input image, For generator, The noise vector is sampled from a Gaussian distribution. The distribution of image data generated by the generator. It is an image that does not match the text. It is an image data distribution that does not match the text. and These are the two hyperparameters of the gradient penalty. It is a norm. It is an operation for finding partial derivatives. This represents the loss of the generator.

[0020] Secondly, in order to achieve the above objectives, the present invention discloses a text-to-image generation system based on spatially aware decoupling, comprising:

[0021] The data input module is used to receive the training dataset and input the training dataset into a pre-established spatial awareness generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatial awareness generative adversarial network model includes a text encoder, a spatial awareness image code mapper, an image generator, and a discriminator.

[0022] The model training module is used to generate enhanced text vectors from text data based on a text encoder, generate layer-specific spatially sensed decoupled image codes from the enhanced text vectors based on a pre-built spatially sensed image code mapper, input the layer-specific spatially sensed decoupled image codes and the enhanced text vectors as conditional inputs to each layer of the image generator to generate text-corresponding images, and input the text-corresponding images into the discriminator for training, with the goal of improving the discrimination ability of image and text matching, and outputting the trained spatially sensed generative adversarial network model;

[0023] The image generation module receives the text to be input, feeds it into the trained spatial awareness generative adversarial network model, and outputs the corresponding image of the input text.

[0024] In another aspect of the present invention, in order to achieve the above-mentioned objective, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs the text-to-image generation method based on spatial awareness decoupling as described above.

[0025] In another aspect of the present invention, in order to achieve the above-mentioned objective, a computer-readable storage medium is disclosed, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is loaded and executed by a processor, it employs the text-to-image generation method based on spatial awareness decoupling as described above.

[0026] The beneficial effects of this invention are:

[0027] This invention transforms low-dimensional text vectors into image codes with spatial dimensions using a spatially aware image code mapper. This serves as an intermediate layer in the text-to-image mapping process, reducing the difficulty of generating complex images and enabling the generator to decode visual semantics before image synthesis, thus improving the generation capability of complex images. Simultaneously, a text conditional code modulator is employed to modulate the image codes element-wise based on text features, effectively extracting spatial visual semantics from the text. The generator's dual fusion module dynamically fuses image features, image codes, and text features from both spatial and channel-aware dimensions through spatial-aware fusion and channel-aware fusion, enhancing the text-image fusion effect. Furthermore, by applying hinge loss and gradient penalty to the discriminator, the adversarial training process is stabilized, prompting the generator to produce images that highly match the text description. Through iterative adversarial training between the generator and discriminator networks, the final model can generate high-fidelity complex images based on text descriptions, exhibiting particularly excellent performance in generating multi-object, multi-shape scenes.

[0028] This invention decouples the text-to-image generation task through a decoupling strategy, including generation process decoupling, hierarchical semantic decoupling, and fusion process decoupling. Generation process decoupling separates the single end-to-end generation into two stages: "generation of text to spatially aware image codes" and "generation of image codes and text to images." The spatially aware image code mapper converts low-dimensional text vectors into spatially aware image codes with spatial dimensions, serving as an intermediate layer in the text-to-image mapping. This separates the semantic decoding and visual synthesis tasks in complex scenes, improving the model's ability and stability in generating multiple objects and complex layouts. Hierarchical semantic decoupling further decouples the global image code into layer-specific image codes. Based on layer-specific adapters, the global image code is mapped to layer-specific image codes at different levels of the corresponding image generator, improving the model's fine-grained control over the generated content. The decoupling of the fusion process decouples the conditions in image generation from the image fusion process into a fusion of spatial and channel dimensions. A dual fusion module modulates the generated image features based on layer-specific image codes and text vectors, respectively, further enhancing the image generator's multi-dimensional fine-grained control over the generated content and improving the fusion effect between text conditions and images. Furthermore, by employing hinge loss and gradient penalty on the discriminator, the adversarial training process is stabilized, prompting the generator to produce images that highly match the text description. Through iterative adversarial training between the generator and discriminator networks, the final model can generate high-fidelity complex images based on text descriptions, especially excelling in generating multi-object, multi-shape scenes. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0031] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0032] Figure 3 This is a schematic diagram of the text conditional code modulator of the present invention;

[0033] Figure 4 This is a schematic diagram of the dual fusion module of the present invention;

[0034] Figure 5 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1:

[0037] like Figure 1 As shown, a text-to-image generation method based on spatially aware decoupling includes the following steps:

[0038] S101: Receive the training dataset and input it into a pre-established spatially aware generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatially aware generative adversarial network model includes a text encoder, a spatially aware image code mapper, an image generator, and a discriminator.

[0039] Each image in the training dataset corresponds to one or more corresponding text descriptions. The text descriptions include the objects, shapes, and spatial relationships in the image. The training dataset includes simple scenes containing a single object and complex scenes containing multiple objects.

[0040] The text encoder employs a pre-trained bidirectional LSTM network, the expression of which is as follows:

[0041]

[0042] Where T is the input text and e is the encoded text vector.

[0043] The generator contains multiple generation blocks and a dual fusion module, and the discriminator contains multiple discriminator blocks and convolutional layers. The dual fusion module incorporates image features based on text information during the image generation process.

[0044] Specifically, to obtain the training dataset for text-image matching, this invention uses two publicly available benchmark datasets: the CUB-200-2011 dataset and the COCO dataset. The CUB-200-2011 dataset contains 11,788 bird images covering 200 bird species, with each image corresponding to 10 text descriptions. It is divided into a training set (150 categories) and a test set (50 categories). The COCO dataset contains 120,000 images of everyday scenes, covering 80 object categories, with each image corresponding to 5 text descriptions. 80,000 images are used as the training set, and 40,000 as the test set. Both datasets contain complex scenes, effectively validating the model's ability to generate complex images.

[0045] A pre-built spatially aware generative adversarial network (GAN) model generates 8×8×64 spatially aware image codes through fully connected layers and five modulation blocks. Each modulation block contains a convolutional layer, a batch normalization layer, a text conditional code modulator, and a ReLU activation function. The generator contains six generation blocks, progressively upsampling image features from 4×4×256 to the target resolution. Each generation block embeds a dual fusion module. The discriminator contains multiple discriminant blocks and convolutional layers, taking an image and text vector as input, and outputting a scalar to determine image authenticity and text matching degree. The text conditional code modulator enhances the text vector to generate scaling and translation parameters, performing element-wise modulation of the image codes. The dual fusion module includes spatially aware fusion and channel-aware fusion, dynamically combining the two fusion results through learnable weights.

[0046] A pre-trained bidirectional long short-term memory (LSTM) network was used as the text encoder, with an embedding layer dimension of 300 and 128 hidden layer neurons. No parameters were updated during training. The text encoding process was as follows: each word in the text was converted into a one-hot code, which was then mapped to a 300-dimensional word vector through the embedding layer; the sequence of word vectors was input into the bidirectional LSM network, and the last hidden state was taken as a 128-dimensional text vector; the text vector was concatenated with a noise vector (128-dimensional) to form an enhanced text vector (256-dimensional), which was used for modulation in the text conditional code modulator and the dual fusion module.

[0047] S102: The text data is used to generate enhanced text vectors based on the text encoder. The enhanced text vectors are used to generate layer-specific spatially sensed decoupled image codes based on a pre-built spatially sensed image code mapper. The layer-specific spatially sensed decoupled image codes and the enhanced text vectors are used as conditional inputs to each layer of the image generator to generate text-corresponding images. The text-corresponding images are input into the discriminator for training, with the goal of improving the discrimination ability of image and text matching. The trained spatially sensed generative adversarial network model is output.

[0048] The spatially aware image code mapper comprises multiple image code modulation modules and layer-specific adapter modules. Taking enhanced text vectors and noise as input, it generates an initial image code through a fully connected layer, which is then refined by several cascaded image code modulation modules to obtain a global spatially aware image code. The layer-specific adapter module further maps the global image code to layer-specific image codes that match the layers of the generator. Existing methods employ direct conditionalization, directly injecting text vectors into the generator, which makes it difficult to explicitly express spatial information, leading to positioning errors or semantic conflicts in complex layout scenarios. The spatially aware image code mapper of this invention decouples the difficult direct text-to-image generation into a text-to-image code and then-to-image generation process, significantly improving the ability to express spatial location and reducing the difficulty for the image generator to learn spatial layout.

[0049] The image generator comprises multiple upsampling generation blocks integrating a dual fusion module. It upsamples low-resolution features layer by layer according to scale to obtain the final image. Each layer receives a layer-specific image code and an enhanced text vector as conditional information from a spatially aware image code mapper. Existing methods employ channel-dimensional affine transformations or attention mechanisms to incorporate text information, making it difficult to balance fusion quality and efficiency. The image generator of this invention, by employing a spatial-channel multi-dimensional fusion strategy, avoids the complex computations of attention mechanisms while maintaining fine-grained control over text and image fusion, thereby improving the fusion effect.

[0050] The discriminator comprises multiple discriminant blocks and convolutional layers. Generated or real images have their features extracted by the convolutional modules. Text vectors are then incorporated and concatenated with the image features before being processed by several convolutional / fully connected layers in the discriminator to output the final score. During training, the discriminator receives three types of pairings: input text with a real image, input text with a generated image, and input text with a mismatched real image. It calculates the loss based on regularization techniques such as hinge loss and gradient penalty to guide generator updates. Existing methods use shallow discriminators, making it difficult to perform adversarial training against spatially aware image code mappers and image generators, resulting in unstable training processes. The discriminator of this invention improves its discriminative ability by deepening the discriminator network, increasing the kernel size of the convolutional layers, employing dilated convolutions, and increasing the number of network channels, thereby achieving overall stability in adversarial training.

[0051] The image code modulation module modulates the initial image code based on text, predicts position-by-position and channel-by-channel spatial and channel-by-channel scaling and translation parameters based on the enhanced text vector, and applies an affine transformation to the initial image code based on the predicted parameters, achieving fine-grained conditionalization and progressive refinement of the image code in both spatial and channel dimensions. Existing methods rely on global conditional vectors or attention mechanisms to implicitly adjust image content in the feature space, lacking explicit spatial mapping. The explicit image code modulation strategy of this invention reduces the probability of semantic and positional confusion, thereby improving multi-object localization and attribute assignment.

[0052] The layer-specific adapter module is used to convert the global image code into layer-specific image codes corresponding to different layers in the image generator. It performs resolution transformation, channel mapping, and necessary convolution processing on the global image code, decoupling it into layer-specific image codes that match the input size and semantic requirements of each layer in the generator. This allows generated blocks at different scales to receive targeted spatial cues and semantic conditions. Existing methods do not display the generated and modulated image codes, resulting in the direct absence of layer-specific adapters for different layers. The layer-specific adapter of this invention controls layout and details at different fine-grained levels, eliminating information mismatch problems caused by cross-scale issues and improving the independent controllability of local and global content.

[0053] The dual fusion module performs spatial and channel-dimensional fusion based on layer-specific image codes and enhanced text vectors, respectively. Within each generation scale, it performs spatial fusion conditioned on layer-specific image codes and channel-dimensional fusion conditioned on enhanced text vectors in parallel, and adaptively integrates the two outputs through learnable weighting or gating mechanisms. Existing methods either modulate only the channels or rely solely on attention for spatial filtering, both of which have limitations when used alone. The dual fusion module of this invention combines the advantages of both, employing lightweight affine operations to achieve channel and spatial modulation, and compensating for their respective shortcomings through adaptive fusion, achieving full and efficient conditional fusion.

[0054] The discriminator's differentiation process includes:

[0055] Input text and real image, input text and generated image, and input text and non-matching real image;

[0056] The discriminator assigns a high score when input text and a real image are input together; a low score when input text and a generated image are input together; and a low score when input text and a mismatched real image are input together. The score ranges from -1 to 1, inclusive, with -1 being the lowest score and 1 being the highest.

[0057] The discriminator employs hinge loss and gradient penalty to stabilize the training process during training.

[0058] The loss function of the discriminator can be expressed as:

[0059]

[0060] in, For the loss of the discriminator, As expected, For input image, For the true distribution of image data, To find the minimum value function, For discriminator, For text vectors that match the input image, For generator, The noise vector is sampled from a Gaussian distribution. The distribution of image data generated by the generator. It is an image that does not match the text. It is an image data distribution that does not match the text.

[0061] By jointly judging the above three types of image-text pairings, the final discriminator can not only distinguish between generated images and real images, but also determine whether an image matches the text, thereby helping the generator to generate real images that match the given text.

[0062] To stabilize the adversarial training process, a hinge loss combined with gradient penalty is employed. The discriminator loss includes the loss calculations for positive and negative samples, as well as a gradient penalty term. The generator loss is the negative value of the discriminator's output for the generated image. The generator minimizes this loss to induce the discriminator to mistake the generated image for a real image and match it with the text. The gradient penalty can be expressed as:

[0063]

[0064] in and These are the two hyperparameters of the gradient penalty. It is a norm. It is an operation for finding partial derivatives. This represents the loss of the generator.

[0065] The expression for the pre-established spatially aware generative adversarial network model is as follows:

[0066]

[0067] in, For the loss of the discriminator, As expected, For input image, For the true distribution of image data, To find the minimum value function, For discriminator, For text vectors that match the input image, For generator, The noise vector is sampled from a Gaussian distribution. The distribution of image data generated by the generator. It is an image that does not match the text. It is an image data distribution that does not match the text. and These are the two hyperparameters of the gradient penalty. It is a norm. It is an operation for finding partial derivatives. This represents the loss of the generator.

[0068] The generator training process consists of three stages. First, image code generation: the noise vector is converted into an initial image code through a fully connected layer of a spatially aware image code mapper, and then processed by five modulation blocks. Each modulation block is modulated by a text conditional code modulator to integrate the code into the spatial semantics of the text, ultimately generating a global image code. The global code is then upsampled to the same size as the image features of each generation block by a layer-specific adapter to obtain a layer-specific image code. Second, feature fusion and image generation: the generator receives a noise vector (converted to 4×4×256 features through a fully connected layer) and upsamples it step by step through six generation blocks. In each generation block, a dual fusion module fuses the current image features with the layer-specific image code and the enhanced text vector. Finally, parameter update: the generator calculates the loss based on the discriminator's output and updates the network parameters using gradient descent (learning rate 0.0001, Adam optimizer, β1=0.0, β2=0.9).

[0069] S103: Receive the text to be input, input the text to be input into the trained spatial awareness generative adversarial network model, and output the corresponding image of the input text.

[0070] After training, inputting text descriptions will generate corresponding images. To verify the effectiveness of this invention, it is compared with existing state-of-the-art models on the CUB and COCO datasets, using Inception Score, Fréchet Inception Distance, and CLIPSIM as evaluation metrics. Experimental results show that on the CUB dataset, the Spatial Aware Generative Adversarial Network (SGAG) achieves an Inception Score (IS) of 5.15 and a Fréchet Inception Distance (FID) of 12.73; on the COCO dataset, it achieves a Fréchet Inception Distance (FID) of 16.55 and a CLIPSIM of 0.298, both outperforming the comparison models and demonstrating the superiority of SGAG in generating complex images.

[0071] Table 1: Comparison results of this method with the other two methods on the CUB and COCO datasets.

[0072]

[0073] like Figure 2 As shown, in the network structure of this invention, the process begins with "text," which serves as input and enters the "text encoder" for processing. After encoding by the text encoder, the output information is transmitted to both the "spatial-aware image code predictor" and the "image generator." The spatial-aware image code predictor receives information from the text encoder, outputs the "predicted image code," and transmits it to the image generator. The image generator generates the "generated image" based on the received information. The generated image is then input to the "discriminator," which processes it and outputs a "score." Specifically, with text as the initial input, the text is first encoded by the text encoder. The encoded information is transmitted in two paths: one path enters the spatial-aware image code mapper, and the other path directly inputs to the image generator. The result processed by the spatial-aware image code mapper is also sent to the image generator, which generates the corresponding image. The generated image then enters the discriminator, which performs authentication and other checks on the generated image, ultimately outputting a score representing the authentication result. This constitutes a complete network process from text input to score output, realizing the functions of text-to-image generation and authentication verification.

[0074] like Figure 3As shown, in the text conditional code modulator, the top inputs are initially a "text vector" and a "noise vector," which together serve as the initial input signals. Starting from the input, the process is divided into two parallel processing branches (distinguished by dashed boxes), with similar structures. Looking at the left branch: after the text and noise vectors are input, they enter a fully connected layer, then pass through an activation function, and then through another fully connected layer. Afterward, a transformation operation is performed, resulting in a matrix output. Simultaneously, the other path of text and noise vectors passes through a fully connected layer, an activation function, and another fully connected layer, resulting in a vector output. These two outputs are multiplied in a multiplication module, and the result is used as a scaling parameter for subsequent scaling operations on the "image code." The processing logic of the right branch is the same as that of the left branch: after the text vector and noise vector are input, they first pass through a "fully connected layer," "activation function," "fully connected layer," and are then "transformed" to obtain a matrix output; another path passes through a "fully connected layer," "activation function," and "fully connected layer" to obtain a vector output. After the two are multiplied in the "multiplication" module, the output is used as a "translation parameter" to perform a "translation operation" on the image code after scaling. Finally, the "image code" first undergoes a scaling operation (based on the scaling parameters obtained from the left branch), and then a translation operation (based on the translation parameters obtained from the right branch) to obtain the "modulated image code," completing the entire process.

[0075] like Figure 4 As shown, in the dual fusion module, there are initially two inputs: "image features" and "image code," as well as "text vector" and "noise vector." The "image code" is passed to the "spatial affine" module as one input; the "image features" provide input to both the "spatial affine" and "channel affine" modules; and the "text vector" and "noise vector" are input to the "channel affine" module. In the "spatial affine" module, the received image code and image features are processed and then subjected to an "activation function" operation; the "channel affine" module receives image features, text vector, and noise vector, processes them, and also undergoes an "activation function" operation. Subsequently, the outputs of both the "spatial affine-activation function" and "channel affine-activation function" paths are input to the "addition" module for fusion calculation. Finally, the output of the "addition" module is the "fused image features," completing the entire feature fusion process.

[0076] Example 2: To achieve the above objective, such as Figure 5 As shown, based on Embodiment 1, this invention discloses a text-to-image generation system based on spatially aware decoupling, comprising:

[0077] The data input module 11 is used to receive the training dataset and input the training dataset into a pre-established spatial awareness generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatial awareness generative adversarial network model includes a text encoder, a spatial awareness image code mapper, an image generator, and a discriminator.

[0078] The model training module 12 is used to generate enhanced text vectors from text data based on a text encoder, generate layer-specific spatially sensed decoupled image codes from the enhanced text vectors based on a pre-built spatially sensed image code mapper, input the layer-specific spatially sensed decoupled image codes and the enhanced text vectors as conditions into each layer of the image generator to generate text-corresponding images, input the text-corresponding images into the discriminator for training, with the goal of improving the discriminative ability of image and text matching, and output the trained spatially sensed generative adversarial network model.

[0079] Image generation module 13 is used to receive the text to be input, input the text to be input into the trained spatial awareness generative adversarial network model, and output the corresponding image of the input text.

[0080] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0081] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0082] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0083] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A text-to-image generation method based on spatially aware decoupling, characterized in that, The method includes the following steps: The training dataset is received and input into a pre-established spatially aware generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatially aware generative adversarial network model includes a text encoder, a spatially aware image code mapper, an image generator, and a discriminator. The spatially aware image code mapper includes multiple image code modulation modules and layer-specific adapter modules. The generator includes multiple upsampling generation blocks integrating a dual fusion module. The discriminator includes multiple discriminator blocks and convolutional layers. The image code modulation modules perform spatial information modulation on the initial image code based on text. The layer-specific adapter module is used to convert the global image code into layer-specific image codes corresponding to different layers in the image generator. The dual fusion module performs spatial and channel-dimensional fusion based on the layer-specific image code and the enhanced text vector, respectively. The text data is used to generate enhanced text vectors based on a text encoder. The enhanced text vectors are then used to generate layer-specific spatially sensed decoupled image codes based on a pre-built spatially sensed image code mapper. The layer-specific spatially sensed decoupled image codes and the enhanced text vectors are used as conditional inputs to each layer of the image generator to generate text-corresponding images. The text-corresponding images are then input into the discriminator for training. The goal is to improve the discriminative ability of image and text matching. The output is the trained spatially sensed generative adversarial network model. The system receives the text to be input, feeds it into the trained spatial awareness generative adversarial network model, and outputs the corresponding image of the input text.

2. The text-to-image generation method based on spatially aware decoupling according to claim 1, characterized in that, Each image in the training dataset corresponds to one or more corresponding text descriptions. The text descriptions include the objects, shapes, and spatial relationships in the image. The training dataset includes simple scenes containing a single object and complex scenes containing multiple objects.

3. The text-to-image generation method based on spatially aware decoupling according to claim 1, characterized in that, The text encoder employs a pre-trained bidirectional LSTM network, the expression of which is as follows: Where T is the input text and e is the encoded text vector, the encoded text vector is concatenated with a random noise vector to obtain an enhanced text vector with increased diversity.

4. The text-to-image generation method based on spatially aware decoupling according to claim 1, characterized in that, The discriminator's differentiation process includes: The input text is matched with a real image, the input text is matched with a generated image, and the input text is matched with a mismatched real image; wherein, the input text is identical, and the mismatched real image is a real image randomly selected after the order of the corresponding text is shuffled. The discriminator assigns a high score when input text and a real image are input together; a low score when input text and a generated image are input together; and a low score when input text and a mismatched real image are input together. The score range is from -1 to 1, inclusive, with -1 being the lowest score and 1 being the highest score.

5. The text-to-image generation method based on spatially aware decoupling according to claim 4, characterized in that, The discriminator employs hinge loss and gradient penalty to stabilize the training process during training.

6. The text-to-image generation method based on spatially aware decoupling according to claim 1, characterized in that, The expression for the pre-established spatially aware generative adversarial network model is as follows: in, For the loss of the discriminator, As expected, For input image, For the true distribution of image data, To find the minimum value function, For discriminator, For text vectors that match the input image, For generator, The noise vector is sampled from a Gaussian distribution. The distribution of image data generated by the generator. It is an image that does not match the text. It is an image data distribution that does not match the text. and These are the two hyperparameters of the gradient penalty. It is a norm. It is an operation for finding partial derivatives. This represents the loss of the generator.

7. A text-to-image generation system based on spatially aware decoupling, employing the text-to-image generation method based on spatially aware decoupling as described in any one of claims 1 to 6, characterized in that... include: The data input module is used to receive the training dataset and input the training dataset into a pre-established spatial awareness generative adversarial network model for training. The training dataset includes multiple samples, and the text data in each sample corresponds to the image data. The pre-established spatial awareness generative adversarial network model includes a text encoder, a spatial awareness image code mapper, an image generator, and a discriminator. The model training module is used to generate enhanced text vectors from text data based on a text encoder, generate layer-specific spatially sensed decoupled image codes from the enhanced text vectors based on a pre-built spatially sensed image code mapper, input the layer-specific spatially sensed decoupled image codes and the enhanced text vectors as conditional inputs to each layer of the image generator to generate text-corresponding images, and input the text-corresponding images into the discriminator for training, with the goal of improving the discrimination ability of image and text matching, and outputting the trained spatially sensed generative adversarial network model; The image generation module receives the text to be input, feeds it into the trained spatial awareness generative adversarial network model, and outputs the corresponding image of the input text.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the text-to-image generation method based on spatial awareness decoupling as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the text-to-image generation method based on spatial awareness decoupling as described in any one of claims 1 to 6.

Images