Text image generation method based on wavelet representation and domain adaptive strategy

Abstract

The invention discloses a text image generation method based on wavelet representation and a domain adaptive strategy. The method comprises the following steps: inputting text description into a pre-trained CLIP text encoder to generate a semantic vector; alternately splicing the semantic vector and the Gaussian noise vector according to elements to generate a fusion vector; after the fusion vector is processed by a multi-layer perceptron, dimension expansion is carried out on the fusion vector through an up-sampling module, and meanwhile high-frequency and low-frequency information is decomposed through wavelet transform; performing enhancement processing on the decomposed high-frequency information, and reconstructing the high-frequency information and the low-frequency information to generate image features; converting the image features into an RGB format to output and generate an image; and inputting the generated image into a discriminator, calculating adversarial loss through an adaptive sample domain constraint strategy, and optimizing the generation process to complete the text generated image. According to the method, the quality of the generated image is improved, and the original semantic information is better maintained.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology, and particularly relates to a text-to-image generation method based on wavelet representation and domain adaptation strategy. Background Technology

[0002] Text-to-image synthesis technology provides a simple and quick approach to the conversion between natural language and images. Belonging to conditional image generation and multimodal learning, its main task is to generate a matching image based on the input descriptive language. Because language possesses expressiveness, logic, standardization, and creativity, text-to-image synthesis is widely used in fields such as computer-aided creation, game development, image editing, and human-computer interaction.

[0003] The key technical challenge in text-to-image generation lies in establishing semantic connections between text and image data, two different modalities, so that the generated image matches the content described in the text. Generative Adversarial Networks (GANs) generate images through adversarial training. They consist of a generator and a discriminator. The generator receives semantic information from natural language and progressively generates matching images. The discriminator judges the authenticity of the images. The two interact adversarially, improving the quality of the generated images. GANs offer fast training speeds, low computational resource requirements, and mature algorithmic frameworks, making them highly competitive under resource-constrained conditions.

[0004] In recent years, many methods have enhanced the consistency between global and local semantic correspondences by incorporating modules such as attention. Some researchers have used stacked network structures to generate necessary details and vivid targets. Others have addressed the weak convergence phenomenon caused by training networks with integral probability indices, focusing on training the intermediate distribution to obtain a new adversarial loss. Still others have combined data augmentation and pre-training with other optimization techniques to improve the quality of images generated by GANs.

[0005] However, the methods described above all rely on high-end hardware and require significant computational resources. Due to the inherent unpredictability and instability of image generation, generating high-quality images with low hardware requirements and computational costs is a pressing issue. Natural language and images belong to different modalities, and significant differences exist between them. Establishing a close correspondence between images and semantic representations while maintaining semantic consistency is another challenge. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a text-to-image generation method based on wavelet representation and domain adaptive strategy. It provides a high-frequency enhancement generator to improve the semantic relevance between the generated image and the text, and an adaptive sample domain constraint strategy to improve the quality of the generated image. In addition, it employs an alternating stitching method to better preserve the original semantic information.

[0007] To achieve the above objectives, this invention provides a text-to-image generation method based on wavelet representation and domain adaptation strategies, comprising:

[0008] Input the text description into the pre-trained CLIP text encoder to generate semantic vectors;

[0009] The semantic vector and the Gaussian noise vector are concatenated element-wise to generate a fused vector;

[0010] The fused vector is processed by a multilayer perceptron and then expanded in dimension by an upsampling module. At the same time, high-frequency and low-frequency information is decomposed by wavelet transform.

[0011] The high-frequency information after decomposition is enhanced and then reconstructed with the low-frequency information to generate image features.

[0012] The image features are converted into RGB format and output to generate an image;

[0013] The generated image is input into the discriminator, and the adversarial loss is calculated through an adaptive sample domain constraint strategy to optimize the generation process and complete the text-to-image generation.

[0014] Optionally, the wavelet transform decomposition process includes:

[0015] The input features are decomposed into low-frequency subbands and high-frequency subbands in three directions using Haar wavelet transform;

[0016] The low-frequency subband retains approximate image information, while the high-frequency subband contains detailed information in the horizontal, vertical, and diagonal directions, respectively.

[0017] Optional high-frequency information enhancement processing includes:

[0018] Reconstruct the high-frequency subbands obtained from the decomposition using inverse wavelet transform;

[0019] The reconstructed high-frequency components are added to the upsampled features via a jump connection.

[0020] Optionally, the upsampling module's processing includes:

[0021] Integrating textual semantic information into image features through affine transformation;

[0022] The affine transformation is achieved through a scaling matrix and a displacement vector, with parameters calculated from the text vector using a multilayer perceptron.

[0023] Optional adaptive sample domain constraint strategies include:

[0024] New samples are generated through linear mixing to expand the sample space;

[0025] The Mixup method was used to perform a second mixing of the samples;

[0026] Calculate the distance between similar samples constrained by the shrinkage loss.

[0027] Optionally, linear mixtures for generating new samples include:

[0028] A new sample is generated by selecting n samples from the same type of sample and calculating their average value.

[0029] Optionally, the discriminator's processing includes:

[0030] The generated image is processed through stacked downsampling modules to extract features;

[0031] Concatenate the downsampled features with the text semantic vector;

[0032] Calculate the adversarial loss function that includes a gradient penalty term.

[0033] Optionally, the upsampling module consists of an upsampling layer and two fusion units stacked together, wherein the fusion unit consists of an affine layer, a ReLU layer, and a convolutional layer.

[0034] Technical Effects of this Invention: This invention discloses a text-to-image generation method based on wavelet representation and domain adaptation strategies. It can generate high-quality images with strong semantic relevance even with limited hardware support. A method of alternating concatenation of text and noise vectors is designed, which better preserves the original semantic information. A high-frequency enhancement generator is proposed, utilizing wavelet transform to improve the model's response to detailed information. Leveraging the sensitivity of wavelet transform to signal abrupt changes, this generator establishes a closer relationship between the image and its semantics. Experiments demonstrate that the wavelet-reconstructed image retains richer and more compact attribute features. Furthermore, an adaptive sample domain constraint strategy is proposed. Through sample domain reconstruction and contraction, the competitive learning between the generator and discriminator is accelerated. This strategy helps the discriminator handle edge and off-center data distributions, avoiding pattern collapse and maintaining the stability of the learning process. Attached Figure Description

[0035] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0036] Figure 1 This is a flowchart illustrating a text-to-image generation method based on wavelet representation and domain adaptation strategy according to an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the text generation image model structure according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the specific structure of upsampling in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram showing the comparison results of images generated on the CUB Birds dataset by embodiments of the present invention and other methods;

[0040] Figure 5 This is a schematic diagram showing the comparison results of images generated on the COCO dataset by embodiments of the present invention and other methods. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0043] like Figure 1 As shown, this embodiment provides a text-to-image generation method based on wavelet representation and domain adaptation strategy, including:

[0044] Input the text description into the pre-trained CLIP text encoder to generate semantic vectors;

[0045] The semantic vector and the Gaussian noise vector are concatenated element-wise to generate a fused vector;

[0046] The fused vector is processed by a multilayer perceptron and then expanded in dimension by an upsampling module. At the same time, high-frequency and low-frequency information is decomposed by wavelet transform.

[0047] The high-frequency information after decomposition is enhanced and then reconstructed with the low-frequency information to generate image features.

[0048] The image features are converted into RGB format and output to generate an image;

[0049] The generated image is input into the discriminator, and the adversarial loss is calculated through an adaptive sample domain constraint strategy to optimize the generation process and complete the text-to-image generation.

[0050] Furthermore, the text-to-image model structure consists of a generation part and a discriminator part, such as... Figure 2 As shown, the upper part of the dashed line generates the image, and the lower part distinguishes between real and fake images.

[0051] The High-Frequency Enhancement Generator (HFE-G) takes two inputs. One is a frozen Vit-B / 32 CLIP-encoded text vector. Compared to traditional text encoders, CLIP learns from 400 million text-image pairs, giving it powerful semantic understanding and representation capabilities across multiple visual tasks. The generation part directly loads CLIP pre-trained weights, requiring only linear projection to encode image semantics. The other input is a noise vector generated by a Gaussian distribution. Using only text as a generation condition leads to a lack of stylistic diversity and limitations in the generated images. The noise vector controls the generation details, producing samples with the same semantics but different visual effects. In summary, text determines semantic relevance, while noise determines stylistic freedom.

[0052] The encoded text and noise vectors are cross-concatenated and then fed together into the upsampling process of HFE-G. During cross-concatenation, the text and noise vectors are concatenated element-wise alternately. The noise vector is then processed separately by a Multilayer Perceptron (MLP) before being reshaped and fed into the upsampling process. This series of upsampling operations completes the fusion of text and image information. Upsampling essentially involves repetitive convolution calculations, which leads to the loss and omission of detailed information. From a signal theory perspective, increasing the number of neural network layers inevitably causes drastic changes in the signal. Among these changes, the high-frequency signals that determine detailed information are more significantly disturbed than the low-frequency signals that determine the basic shape. The high-frequency enhancement module uses wavelet transform to perform multi-scale analysis of the signal, simultaneously obtaining time and frequency domain information, which is beneficial for stable signals. The feature map generated by HFE-G is then processed using RGB operations to obtain the image.

[0053] In the discrimination section, the image obtained by the generator is fed to the Target-Aware Discriminator (TAW-D). After stacked downsampling, it is concatenated with the text vector to obtain image features. Traditional discriminators widely publish training data in the feature space, which leads to ambiguous judgments during the learning process and inappropriately providing high confidence for biased data. Based on this consideration, this invention adopts an Adaptive Sample Domain Constraint (ASDC) strategy. ASDC mainly includes two processes: adaptive sample reconstruction and feature space shrinkage. The relationships between the original sampling points in the sample domain are not close enough, causing a bias in the distribution of confidence. Therefore, adaptive sample reconstruction uses data augmentation methods to adaptively process the samples in the sampling space into new samples. The new samples are more closely related to the old samples, thus constructing a larger sample domain space. A loss function is designed in the new sample domain space, and the feature space is gradually shrunk during training. The ASDC strategy improves the learning quality of the discriminator by constraining the training region. The overall process is controlled by adversarial loss adjustment.

[0054] High-frequency enhancement generator:

[0055] HFE-G mainly consists of upsampling and high-frequency enhancement. The specific structure of the upsampling is as follows: Figure 3 As shown, it mainly consists of upsampling and two stacked fusion units. The fusion unit consists of an affine layer, a ReLU layer, and a convolutional layer. The affine transformation, which only involves matrix calculations, can preserve the information of the input data to the greatest extent and map data of different dimensions to a fixed dimension, thus achieving data format unification.

[0056] Input upsampled data After upsampling and scaling, the result is... Where S c This is the scaling factor for upsampling.

[0057] The upsampled feature map is then fed into the affine layer. For a given feature map X, the affine transformation... This is achieved using the scaling matrix c and the displacement vector β.

[0058]

[0059] X here i C is a matrix representing the i-th channel of the feature map. i and β i The word embedding vector e is used to calculate C = MLP1(e) and β = MLP2(e) using a multilayer perceptron.

[0060] To expand the representation space and enable the generator to produce richer visual effects from the text, it is essential to add ReLU layers between affine layers.

[0061] Therefore, the generator with upsampling stacks can not only fully integrate text information, but also support the generation of semantically consistent images from different text descriptions.

[0062] Wavelet transform can be applied to signal denoising, compression, and filtering to analyze unstable signals. In image processing and machine vision, wavelet transform is commonly used for edge detection. The key to this technique is signal separation and reconstruction. Unlike Fourier transform, wavelet transform uses wavelet coefficients to decompose the signal, obtaining signals at different frequencies and locations.

[0063] Haar wavelet transform is widely used due to its simple algorithm and ease of implementation. It has two specific operations: Discrete Wavelet Transform (DWT) and Inverse Wavelet Transform (IWT). DWT uses low-pass and high-pass filters to decompose the original image signal into approximate and detail coefficients, which can represent the hierarchical details of image information. IWT accurately reconstructs the original signal by recursively synthesizing the subbands of each layer, restoring the information loss during wavelet decomposition. Haar consists of two inverse operations: wavelet pooling and wavelet depooling. Let... and These represent low-pass and high-pass filters, respectively. This leads to the construction of four kernels (LL) in wavelet pooling. T LH T HL T and HH T These represent a low-pass filter, a horizontal high-pass filter, a vertical high-pass filter, and a diagonal high-pass filter, respectively. The Haar wavelet transform recursively decomposes the input feature X into four sub-bands;

[0064]

[0065] Where * represents convolution calculation. The low-frequency subband obtained from X decomposition is obtained through cascaded transformation and filtering. It retains approximate information of the input image, high-frequency subband. and These represent detailed information in the horizontal, vertical, and diagonal directions, respectively.

[0066] In HFE-G, this Haar wavelet transform is used to decompose the input signal, thereby enhancing high-frequency information. The frequency components of the i-th layer features obtained from the Haar transform are... and Perform depooling operations.

[0067]

[0068] Among them, X i+1 X' represents the wavelet pooling operation of the (i+1)th layer. i+1 It's a redirect link.

[0069] Target perception and discriminator:

[0070] The discriminator converts the image into image features through a series of downsampling steps. These image features are then concatenated with sentence vectors, and a discriminative loss is used to evaluate the consistency between the image and the semantics.

[0071] The discrimination results can be categorized into four types: semantic match with the synthetic image, semantic mismatch with the synthetic image, semantic match with the real image, and semantic mismatch with the real image. Based on these results, the optimization objective of the discriminator can be set as follows:

[0072]

[0073] in, From the real data distribution I real The expected value of the sampled data. From the generated prior distribution I gene The expected value of the sampling noise in the middle. Indicates from I mis The expected value of the sampled X is used to enhance the generalization ability of the discrimination. ||.|| is the L2 norm of the output gradient.

[0074] Adaptive sample domain constraints:

[0075] ASDC consists of two steps: adaptive sample reconstruction and feature space shrinkage. The discriminator's workflow can be viewed as a classification process. Traditional classifiers indiscriminately process samples in the feature space. This learning method comprehensively summarizes the characteristics of the sample distribution, but also assigns high attention to non-mainstream samples. This makes it difficult for the classification process to focus on the correct image distribution. To concentrate on learning the true image distribution, the sample space needs to be shrunk. The result of the shrinkage should fully describe the mainstream distribution patterns to promote the discriminator's learning. Considering these factors, ASDC first reconstructs the samples, expanding the sample space while ensuring the relationships between samples. To meet these requirements, this invention uses data augmentation to synthesize new samples. n samples from the same class in the sample domain are taken and mixed to form a new sample.

[0076]

[0077] This operation involves only linear transformations.

[0078] To generate further samples closely related to the synthesized image, a secondary mixing process using Mixup is employed.

[0079] x ij =h(x i ,x j ,η)=ηx i +(1-η)x j ;

[0080] Where η is a hyperparameter sampled from the sample domain. x ij From training samples x i and The convex combination is generated and projected into the feature space. ij With x i and They belong to the same category, but are more difficult to classify.

[0081] The expanded sample space has x i , and x ij Three samples of the same type but from different sources. A shrinkage loss was used to reduce the distance between the samples.

[0082]

[0083] Here, ∈ represents the noise term from a Gaussian distribution. It is an evaluation metric and a distance function, and the L1 norm or L2 norm is commonly used.

[0084] An application example of this invention is shown below:

[0085] The acquired datasets include the Caltech-UCSD Birds-200-2011 (CUB Birds) and the Microsoft Common Objects in Context (COCO) dataset. CUB Birds contains many fine-grained features, effectively evaluating the model's ability to generate details. This dataset contains 11,788 images covering 200 bird species. For the text-to-image task, each image has descriptions in ten languages ​​to reflect the richness of textual information and the diversity of perspectives. These images primarily focus on the birds themselves, with relatively simple backgrounds. The COCO dataset contains interactions of various everyday objects in complex scenes. Its rich categories and broad content help evaluate the model's generalization ability. This dataset contains 80k training images and 40k test images. Each image has descriptions in five languages ​​to provide a more open and diverse scene understanding.

[0086] This method was performed on a server with a single NVIDIA GeForce RTX 3090 24GB GPU. To generate better images, the images corresponding to the text were resized to 256×256. To evaluate the quality and semantic relevance of the generated images, this invention uses FID (Frechet Inception Distance) and CS (Cosine Similarity) as evaluation metrics. FID evaluates the degree of similarity between the generated sample and the real sample in the feature space. A lower score indicates that it is more difficult to distinguish between real and fake samples at the feature level, and the quality of the generated sample is higher. Its calculation is based on Gaussian distribution fitting. CS evaluates the semantic consistency between text and images by calculating the cosine similarity between text features and image features. A higher CS score indicates that the generated image is more similar to the text at the semantic feature level. Furthermore, the number of parameters is also an evaluation metric worth comparing and discussing.

[0087] The text-to-image generation model mainly consists of a generator and a discriminator. The generator, based on the input image and noise, uses high-frequency enhancement to accurately handle spatial details and local structures in the generated image. The noise is responsible for the randomness of the generated image. The discriminator is used to distinguish between the generated image and the real image.

[0088] To demonstrate the effectiveness of the CLIP text encoder, high-frequency enhancement, and adaptive sample domain constraints, this invention removes these modules while keeping the remaining steps and structure unchanged. The comparative experimental structure for the CUB Birds dataset is shown in Table 1. The model using the CLIP text encoder, high-frequency enhancement, and adaptive sample domain constraints achieves an FID of 12.42 and a CS of 0.2923. The comparative experiments show that the CLIP text encoder and high-frequency enhancement are beneficial for improving semantic consistency. Adaptive sample domain constraints are beneficial for improving image quality.

[0089] Table 1

[0090]

[0091] The proposed invention patent scheme will be compared with the most representative works AttnGAN, DM-GAN and the latest achievements DF-GAN, Adam-GAN and DSE-GAN.

[0092] Table 2 compares the parameter counts of recently popular large-scale models in academia and industry. These models achieve more competitive performance through a larger number of parameters. Among them, the smallest autoregressive model, Muse-3B, requires 3000M parameters, while the smallest diffusion model, GLIDE, requires 5000M. While larger models offer impressive performance, they also increase deployment difficulty on the user end and limit personalized customization. They typically require training on dozens or even hundreds of A100 GPUs. In contrast, the parameter count of this invention is reduced exponentially, requiring only 50.94M parameters, and can be trained on a single 3090 GPU. Notably, on a single 3090 GPU, this invention can synthesize an image in just 0.05 seconds.

[0093] Table 2

[0094]

[0095] Table 3 records the comparison results on the CUB Birds dataset. The results show that, compared to the single-level backbone DF-GAN, the FID of this invention is improved from 14.81 to 12.42, and the CS is improved from 0.2920 to 0.2923. This indicates a significant advantage in generation quality, while maintaining a similar level in category semantic consistency. Compared to LAFITE and SSA-GAN, this invention has a lead of 2 to 3 points. Compared to the diffusion model VQ-Diffusion, this invention is also significantly competitive. Compared to DSE-GAN and KD-GAN, this method has a lead of 1 to 2 points.

[0096] Table 3

[0097]

[0098]

[0099] Figure 4 This section shows the performance of the three algorithms on the CUB Birds dataset. The captions for the images in the CUB Birds dataset are relatively simple, generally consisting of the color corresponding to the subject and other specific adjectives. To facilitate evaluation of the semantic consistency and detail representation of the generated images, the original captions are shown in the last line. Blue text represents color, and red text represents other concrete descriptive words.

[0100] Depend on Figure 4It can be seen that DF-GAN, SSA-GAN, and the present invention can generally satisfy the consistency of image and text semantics. However, the present invention has an inherent advantage in terms of details. As can be seen from the first and second columns, the images generated by the present invention are more natural and realistic. DF-GAN and SSA-GAN are more like works of art created through imagination. As can be seen from the fifth, sixth, and seventh columns, the images generated by the present invention have better foreground and background blending, without any abrupt or confusing foreground and background combinations. As can be seen from the fourth column, even in synthesized images with similar foreground and background colors, the present invention can still handle fine-grained information such as feather color, limbs, and edges well. As can be seen from the fourth, fifth, and sixth columns, the generation of the present invention is not a simple stacking of blurry shapes, such as small areas like eyes, head, and feather shapes. In summary, the bird shapes synthesized by the method of the present invention are smooth, with detailed feathers and consistent semantics.

[0101] Table 4 presents the comparative results on the COCO dataset. Compared with classic models AttnGAN, DM-GAN, and MirrorGAN, this invention has significant advantages. Compared with recent studies DF-GAN and SSA-GAN, the FID of this invention is improved from 19.32 and 19.37 to 16.34, respectively. The performance on the COCO dataset indicates that this invention performs satisfactorily in multi-class target and image synthesis.

[0102] Table 4

[0103]

[0104] Figure 5This is a visualization of images generated on the COCO dataset by DF-GAN, SSA-GAN, and this invention. The images selected for comparison are the same as those from CUB Birds; the best-performing image was chosen from ten generated images. The COCO dataset contains a wide variety of subjects, broadly categorized as living and non-living. Once the subject is determined, semantic consistency can be reflected by adjectives. The last row of the image provides the original caption. Blue text represents living subjects; their unique behaviors and actions are key to ensuring image-text consistency. They serve as foreground in most image synthesis results. Red text represents non-living subjects, serving as the background and primarily describing the scene. The first and sixth columns show human generation samples. Human generation is a challenging problem on the COCO dataset. The surfer and woman generated by this invention have complete morphology, and their actions and expressions are semantically consistent, without distortion or deformation. The background generation is also quite detailed. The second column shows animal generation samples, from which we can see that this invention roughly depicts their outlines and colors with smooth lines. The fourth, fifth, and seventh columns show scene generation. Overall, the scenes synthesized by this invention have a significant advantage in detail.

[0105] This invention proposes a lightweight structure that utilizes a domain-adaptive strategy to reconstruct and shrink the sampling space to improve the quality of generated images. Wavelet transform is used to enhance high-frequency information, thereby improving the semantic consistency of the generator. Furthermore, a pre-trained CLIP encoding is introduced to further align the semantic consistency of text and images. To ensure the randomness of the generated images, semantic features and noise are cross-concatenated. In the generation process, firstly, the text is encoded using a CLIP text encoder, and the encoded vector is cross-concatenated with the noise vector. Secondly, the noise vector undergoes upsampling to increase its dimensionality. During upsampling, it is combined with the concatenated vector, and a high-frequency enhancement module is used to improve semantic relevance. Thirdly, the noise vector undergoes multiple upsampling steps to obtain image features, which are then adjusted to RGB format to obtain the generated image. In the discrimination stage, the generated image undergoes multiple downsampling steps and is combined with the concatenated vector to obtain image features. After processing the label image to obtain image features, important features of the label image are extracted through adaptive sample reconstruction and feature space shrinking. The important features of the label image and the image features of the generated image are used to calculate an adversarial loss, thereby training the generator.

[0106] This invention discloses a text-to-image generation method based on wavelet representation and domain adaptation strategies, which can generate high-quality images with strong semantic relevance even with low hardware support. A method of alternating concatenation of text and noise vectors is designed, which better preserves the original semantic information. A high-frequency enhancement generator is proposed, utilizing wavelet transform to improve the model's response to detailed information. Leveraging the sensitivity of wavelet transform to signal abrupt changes, this generator establishes a closer relationship between the image and its semantics. Experiments demonstrate that the wavelet-reconstructed image retains richer and more compact attribute features. Furthermore, an adaptive sample domain constraint strategy is proposed, accelerating the competitive learning between the generator and discriminator through sample domain reconstruction and contraction. This strategy helps the discriminator handle edge and off-center data distributions, avoiding pattern collapse and maintaining the stability of the learning process.

[0107] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text-to-image generation method based on wavelet representation and domain adaptation strategy, characterized in that, include: Input the text description into the pre-trained CLIP text encoder to generate semantic vectors; The semantic vector and the Gaussian noise vector are concatenated element-wise to generate a fused vector; The fused vector is processed by a multilayer perceptron and then expanded in dimension by an upsampling module. At the same time, high-frequency and low-frequency information is decomposed by wavelet transform. The high-frequency information after decomposition is enhanced and then reconstructed with the low-frequency information to generate image features. The image features are converted into RGB format and output to generate an image; The generated image is input into the discriminator, and the adversarial loss is calculated through an adaptive sample domain constraint strategy to optimize the generation process and complete the text-to-image generation.

2. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 1, characterized in that, The wavelet transform decomposition process includes: The input features are decomposed into low-frequency subbands and high-frequency subbands in three directions using Haar wavelet transform; The low-frequency subband retains approximate image information, while the high-frequency subband contains detailed information in the horizontal, vertical, and diagonal directions, respectively.

3. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 1, characterized in that, High-frequency information enhancement processing includes: Reconstruct the high-frequency subbands obtained from the decomposition using inverse wavelet transform; The reconstructed high-frequency components are added to the upsampled features via a jump connection.

4. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 1, characterized in that, The upsampling module's processing includes: Integrating textual semantic information into image features through affine transformation; The affine transformation is achieved through a scaling matrix and a displacement vector, with parameters calculated from the text vector using a multilayer perceptron.

5. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 1, characterized in that, Adaptive sample domain constraint strategies include: New samples are generated through linear mixing to expand the sample space; The Mixup method was used to perform a second mixing of the samples; Calculate the distance between similar samples constrained by the shrinkage loss.

6. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 5, characterized in that, Linear mixture generation of new samples includes: A new sample is generated by selecting n samples from the same type of sample and calculating their average value.

7. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 1, characterized in that, The discriminator's processing includes: The generated image is processed through stacked downsampling modules to extract features; Concatenate the downsampled features with the text semantic vector; Calculate the adversarial loss function that includes a gradient penalty term.

8. The text-to-image generation method based on wavelet representation and domain adaptation strategy as described in claim 4, characterized in that, The upsampling module consists of an upsampling layer and two fusion units stacked together. The fusion unit consists of an affine layer, a ReLU layer, and a convolutional layer.

Images