Text synthesis image method based on variable mask
By using a variable mask-based multimodal image generation model (MCG) combined with VQ-GAN and CLIP models, image and text information is transformed into semantic tags. By employing a variable masking rate and padding mechanism, the problems of slow generation speed, high computational resources, and low image quality in existing technologies are solved, achieving efficient and diversified image generation and restoration.
Patent Information
- Application Number
- CN202511053897.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-12-05
AI Technical Summary
Existing text-to-image synthesis techniques have shortcomings in terms of generation speed, computational resource consumption, semantic consistency, and image quality. In particular, diffusion models, autoregressive models, and generative adversarial networks have problems such as high hardware requirements, slow generation speed, low image quality, and insufficient diversity.
We employ a variable mask-based multimodal image generation model (MCG) combined with VQ-GAN and CLIP pre-trained models to transform image and text information into unified semantic labels. We use a variable masking rate strategy and a learnable padding mechanism, process long sequence inputs through a ViT encoder-decoder structure, and optimize the model using reconstruction training and contrastive loss to reduce computational cost and memory consumption.
It improves the accuracy and diversity of image generation, enhances the generation and computational efficiency of the model, and is suitable for diverse image generation and image restoration tasks.
Smart Images
Figure CN121074170A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to a method for text-to-image synthesis based on variable masks. Background Technology
[0002] Currently, existing technologies in the field of text-to-image synthesis are mainly based on methods such as diffusion models, autoregressive models, and generative adversarial networks (GANs). While these technologies have made some progress in generating high-quality images, they still have significant shortcomings, especially in terms of generation speed, computational resource consumption, semantic consistency modeling, and image quality.
[0003] The diffusion model generates images by progressively adding noise, eventually transforming the image into Gaussian noise, and then recovering the original image through a denoising process. While diffusion models can generate high-fidelity images, their training and generation processes are computationally expensive, involve a large number of parameters, are slow, and have extremely high hardware requirements. For example, frameworks based on diffusion models, such as DALL-E, require 180,000 hours to train on a single A100 GPU, and even larger models (such as Parti) require over 1 million hours, making the computational cost extremely high. Due to the progressive nature of its generation process, the image generation time is also long, making it difficult to meet the needs of real-time applications.
[0004] The method combining an autoregressive model with a VQ-VAE (Vector Quantization Variational Autoencoder) first converts text information into tokens, then uses a discretized autoencoder (AE) to convert the image back into tokens. These two tokens are then concatenated and input into a generative model for image generation. While this method can efficiently generate images, it suffers from a large number of model parameters, resulting in significant computational overhead and training time, leading to slow generation speeds. Especially in applications involving large, general-purpose models, this method has high hardware requirements, often requiring powerful computing devices to run, which limits its applicability.
[0005] The combination of Generative Adversarial Networks (GANs) and CLIP (Contrastive Language-Image Pre-trained Model) utilizes an adversarial training process between a generator and a discriminator to generate images. The generator produces images based on text input and random noise, while the discriminator judges the realism of the images. Although this approach improves image generation quality to some extent, especially in terms of text-image semantic alignment, it still faces several challenges. First, the lack of channel-level interaction during image synthesis results in insufficient quality and detail in the synthesized images. Second, existing GAN methods are prone to instability during training, and the generated images often lack diversity and detail, making it difficult to generate images that meet expectations in different contexts. Furthermore, GANs also suffer from pattern collapse, leading to a lack of diversity and realism in the generated images.
[0006] While these existing technologies have made progress in image generation to some extent, they still have many shortcomings in terms of generation speed, computational resource requirements, semantic consistency, and the quality and diversity of generated images, which limit their wide applicability and effectiveness in practical applications.
[0007] Therefore, a new solution is needed to address the above problems. Summary of the Invention
[0008] The purpose of this invention is to provide a text-to-image synthesis method based on variable masks to solve the technical problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a text-to-image synthesis method based on variable masks, comprising at least the following steps:
[0010] S1: Build a multimodal image generation model, namely the MCG model. The MCG model transforms image and text information into unified semantic labels through VQ-GAN and CLIP pre-trained models, realizing the effective fusion of visual and text information and laying the foundation for subsequent processing.
[0011] S2: Information compression and reconstruction of the MCG model employs a variable masking rate strategy to randomly mask some semantic labels, forcing the model to learn to reconstruct the complete content from partial information, thereby improving the model's understanding ability and generation efficiency.
[0012] S3: Employ a learnable padding mechanism to further optimize the integrity of the input label sequence;
[0013] S4: Efficient feature processing. The MCG model adopts the ViT encoder-decoder structure, which can effectively process long sequence inputs, capture global semantic information, and improve the model's ability to understand and generate image content.
[0014] S5: Optimize the training strategy by using a reconstruction training mechanism and contrastive loss optimization to improve the semantic consistency and feature representation ability of the model. The reconstruction loss ensures that the model can accurately reconstruct the masked information, while the contrastive loss enhances the model's ability to distinguish between different samples.
[0015] S6: Improved computational efficiency. The masked label removal mechanism significantly reduces computational load and memory consumption by reducing the number of labels that need to be processed, thereby improving training efficiency.
[0016] Furthermore, S1 includes at least the following steps:
[0017] Under VQ-GAN encoding, the input image is first encoded into a set of discrete visual features, which reflect different semantic information in the image, i.e., the semantic label of the image is obtained.
[0018] The CLIP pre-trained model uses the pre-trained visual-language joint embedding space to map the corresponding text information to the same semantic space, thereby obtaining the semantic representation of the text, that is, obtaining the semantic label of the text.
[0019] The semantic labels of images and text are aligned in the same space in this way, providing a good semantic foundation for subsequent processing;
[0020] By mixing semantic tags of images and text, and manipulating the mixed tags, the connection between images and text can be enhanced, thereby improving the accuracy and diversity of generated images.
[0021] The mixed semantic label sequence will be used as the input to the model, and then the labels will be masked according to a randomly selected masking rate.
[0022] Furthermore, the variable masking rate strategy involves randomly selecting a masking rate R from a truncated Gaussian distribution centered at 0.55, with a left truncation of 0.5 and a right truncation of 1.
[0023] The masking rate R determines what percentage of the labels will be randomly masked;
[0024] The masked label is replaced with a learnable "fake" label [M], which is continuously optimized during training to help the model better fill in the masked parts;
[0025] Masking operations can not only reduce information redundancy, but also effectively improve training efficiency because they reduce the number of labels in the input data, thereby reducing computation and memory consumption.
[0026] Furthermore, the learnable filling mechanism fills the gaps in the masked label portion by introducing a learnable "fake" label [C0].
[0027] Furthermore, to ensure the integrity of information flow, the MCG model uses a ViT encoder-decoder structure to process the edited tag sequence;
[0028] The ViT encoder takes the masked label sequence as input and maps it into the latent feature space. The ViT encoder can not only extract information from local label features, but also summarize the features of the image in a global scope.
[0029] The category label [C] of each image acts as a representation of visual features in this process, helping the MCG model understand the image content from a global perspective;
[0030] The latent feature sequences obtained by the ViT encoder will undergo further processing by the decoder;
[0031] Before decoding, the MCG model first pads the ViT encoder output with the category label [C] to restore it to the same length as the original input;
[0032] The decoder reconstructs the image and text semantic labels based on these padded features, gradually restoring the complete information.
[0033] This reconstruction process ensures that the model can generate images that meet expectations and can effectively restore the relationship between textual semantics and visual semantics.
[0034] Furthermore, in the reconstructed training mechanism:
[0035] make Let N represent the potential label sequence obtained from the input image and text annotations, where N is the length of the label sequence;
[0036] make This represents a binary mask, used to determine which tags are masked;
[0037] The training objective of the MCG model is to reconstruct the masked labels from the unmasked labels. Therefore, a cross-entropy loss function is added between the actual one-hot labels and the decoder output, specifically defined as:
[0038]
[0039] Among them, Y M It is the unmasked subset of the label sequence Y, p(y i |Y M ) is based on the unmasked label Y M Label probability prediction under encoder-decoder network conditions, For datasets Expect all samples Y in ;
[0040] This invention optimizes only the loss for the masked labels, because optimizing the loss for all labels simultaneously would degrade the performance of generative learning. Furthermore, a contrastive loss is incorporated to further enhance the model's learning performance.
[0041] The formula for calculating the contrast loss is as follows:
[0042]
[0043] Where z represents the feature after two layers of MLP normalization, B represents the batch size, and τ is the temperature parameter. i , Enhanced view from the same image, negative sample z j These are from other samples in the same batch;
[0044] In this way, contrastive loss is introduced to optimize representation learning and improve the model's performance in multimodal information generation;
[0045] The final training objective loss function is a weighted sum of the reconstruction loss and the contrastive loss, expressed as:
[0046]
[0047] Here, λ = 0.1 is used to balance the ratio of the two losses. The reconstruction loss itself acts as a regularization mechanism, effectively preventing the encoder from learning shortcut solutions, thereby avoiding overfitting and improving the model's generalization ability.
[0048] By combining reconstruction training and contrastive loss, this invention not only optimizes the model's ability to reconstruct masked labels, but also further enhances the model's feature learning ability, promoting the effective alignment of visual semantics and textual semantics.
[0049] Furthermore, the masking label deletion mechanism involves not only masking the labels but also randomly deleting a portion of them. This approach effectively reduces computational load while promoting the model's focused learning of key information, thereby accelerating the training process and improving the quality of generated images. Especially when the masking ratio R is set above 0.5, deleting a large number of masking labels can significantly reduce pre-training time and memory consumption, improving overall training efficiency.
[0050] Compared with the prior art, the beneficial effects of the present invention are:
[0051] This invention proposes a multimodal image generation model (MCG) based on visual and textual information. This model combines VQ-GAN and CLIP, uses a variable masking rate strategy to convert input images and text into semantic labels, and randomly masks some labels to enhance the model's ability to recover missing information. At the same time, learnable "false" labels are introduced to fill the masking positions, narrowing the gap between visual and textual semantics. MCG adopts a ViT architecture encoder-decoder structure to convert mixed labels into latent features, and achieves image and textual information fusion by reconstructing labels in the decoding stage.
[0052] The method of the present invention, based on the above model, can effectively improve the fit between visual and textual semantics in multimodal image generation tasks, while enhancing the model's understanding and generation capabilities of image content. It is applicable to diverse image generation, image restoration, and other tasks. Attached Figure Description
[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a schematic diagram of the entire invention. Detailed Implementation
[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0056] This invention designs an efficient and simple architecture for text-to-image synthesis tasks. By introducing a variable masking rate, it enhances the semantic consistency between text description and image generation. It solves the problems of low-quality, unrealistic-looking, and structurally unreasonable synthesized images in text-to-image synthesis tasks; it also addresses the issue of low semantic consistency between synthesized images and given text information; and it further solves the problem of poor generalization ability of text-to-image synthesis models. Specifically:
[0057] Please see Figure 1 A text-to-image synthesis method based on variable masks includes at least the following steps:
[0058] S1: Build a multimodal image generation model, namely the MCG model. The MCG model transforms image and text information into unified semantic labels through VQ-GAN and CLIP pre-trained models, realizing the effective fusion of visual and text information and laying the foundation for subsequent processing.
[0059] VQ-GAN (Vector Quantized Generative Adversarial Network) is introduced as the image generation component of this invention. VQ-GAN combines the advantages of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), mapping images to a discrete latent space through quantization techniques and optimizing image quality through generative adversarial mechanisms. In this invention, VQ-GAN maps input textual and visual information into latent vectors via a specific encoder; these latent vectors are then used to reconstruct high-resolution images in the decoder. VQ-GAN's advantage lies in its ability to generate detailed and visually realistic images, making it particularly suitable for image generation tasks. Combining this image generation capability, CLIP (Contrastive Language-Image Pre-training) plays a crucial role as another module in this invention. CLIP learns from a large number of text-image pairs through contrastive learning, enabling it to understand the semantic relationships between text and images. In this invention, CLIP is primarily responsible for converting input text into semantic labels and providing semantic guidance during image generation. CLIP's contrastive learning mechanism ensures that the generated images are consistent with the input text description, guaranteeing a high semantic match between text and images.
[0060] S1 includes at least the following steps:
[0061] Under VQ-GAN encoding, the input image is first encoded into a set of discrete visual features, which reflect different semantic information in the image, i.e., the semantic label of the image is obtained.
[0062] The CLIP pre-trained model uses the pre-trained visual-language joint embedding space to map the corresponding text information to the same semantic space, thereby obtaining the semantic representation of the text, that is, obtaining the semantic label of the text.
[0063] The semantic labels of images and text are aligned in the same space in this way, providing a good semantic foundation for subsequent processing;
[0064] By mixing semantic tags of images and text, and manipulating the mixed tags, the connection between images and text can be enhanced, thereby improving the accuracy and diversity of generated images.
[0065] The mixed semantic label sequence will be used as the input to the model, and then the labels will be masked according to a randomly selected masking rate.
[0066] S2: Information compression and reconstruction of the MCG model employs a variable masking rate strategy to randomly mask some semantic labels, forcing the model to learn to reconstruct the complete content from partial information, thereby improving the model's understanding ability and generation efficiency.
[0067] The variable masking rate strategy involves randomly selecting a masking rate R from a truncated Gaussian distribution centered at 0.55, with a left truncation of 0.5 and a right truncation of 1.
[0068] The masking rate R determines what percentage of the labels will be randomly masked;
[0069] The masked label is replaced with a learnable "fake" label [M], which is continuously optimized during training to help the model better fill in the masked parts;
[0070] Masking operations can not only reduce information redundancy, but also effectively improve training efficiency because they reduce the number of labels in the input data, thereby reducing computation and memory consumption.
[0071] S3: Employ a learnable padding mechanism to further optimize the integrity of the input label sequence;
[0072] The learnable filling mechanism fills the gaps in the masked label section by introducing a learnable "fake" label [C0].
[0073] S4: Efficient feature processing. The MCG model adopts the ViT encoder-decoder structure, which can effectively process long sequence inputs, capture global semantic information, and improve the model's ability to understand and generate image content.
[0074] To ensure the integrity of information flow, the MCG model uses a ViT encoder-decoder structure to process the edited tag sequence;
[0075] The ViT encoder takes the masked label sequence as input and maps it into the latent feature space. The ViT encoder can not only extract information from local label features, but also summarize the features of the image in a global scope.
[0076] The category label [C] of each image acts as a representation of visual features in this process, helping the MCG model understand the image content from a global perspective;
[0077] The latent feature sequences obtained by the ViT encoder will undergo further processing by the decoder;
[0078] Before decoding, the MCG model first pads the ViT encoder output with the category label [C] to restore it to the same length as the original input;
[0079] The decoder reconstructs the image and text semantic labels based on these padded features, gradually restoring the complete information.
[0080] This reconstruction process ensures that the model can generate images that meet expectations and can effectively restore the relationship between textual semantics and visual semantics.
[0081] S5: Optimize the training strategy by using a reconstruction training mechanism and contrastive loss optimization to improve the semantic consistency and feature representation ability of the model. The reconstruction loss ensures that the model can accurately reconstruct the masked information, while the contrastive loss enhances the model's ability to distinguish between different samples.
[0082] In reconstructing the training mechanism:
[0083] make Let N represent the potential label sequence obtained from the input image and text annotations, where N is the length of the label sequence;
[0084] make This represents a binary mask, used to determine which tags are masked;
[0085] The training objective of the MCG model is to reconstruct the masked labels from the unmasked labels. Therefore, a cross-entropy loss function is added between the actual one-hot labels and the decoder output, specifically defined as:
[0086]
[0087] Among them, Y M It is the unmasked subset of the label sequence Y, p(y i |Y M ) is based on the unmasked label Y M Label probability prediction under encoder-decoder network conditions, For datasets Expect all samples Y in ;
[0088] This invention optimizes only the loss for the masked labels, because optimizing the loss for all labels simultaneously would degrade the performance of generative learning. Furthermore, a contrastive loss is incorporated to further enhance the model's learning performance.
[0089] The formula for calculating the contrast loss is as follows:
[0090]
[0091] Where z represents the feature after two layers of MLP normalization, B represents the batch size, and τ is the temperature parameter. i , Enhanced view from the same image, negative sample z j These are from other samples in the same batch;
[0092] In this way, contrastive loss is introduced to optimize representation learning and improve the model's performance in multimodal information generation;
[0093] The final training objective loss function is a weighted sum of the reconstruction loss and the contrastive loss, expressed as:
[0094]
[0095] Here, λ = 0.1 is used to balance the ratio of the two losses. The reconstruction loss itself acts as a regularization mechanism, effectively preventing the encoder from learning shortcut solutions, thereby avoiding overfitting and improving the model's generalization ability.
[0096] By combining reconstruction training and contrastive loss, this invention not only optimizes the model's ability to reconstruct masked labels, but also further enhances the model's feature learning ability, promoting the effective alignment of visual semantics and textual semantics.
[0097] S6: Improved computational efficiency. The masked label removal mechanism significantly reduces computational load and memory consumption by reducing the number of labels that need to be processed, thereby improving training efficiency.
[0098] The masking label removal mechanism involves not only masking the labels but also randomly deleting a portion of them. This approach effectively reduces computational cost while promoting the model's focused learning of key information, thereby accelerating the training process and improving the quality of generated images. Especially when the masking ratio R is set above 0.5, deleting a large number of masking labels can significantly reduce pre-training time and memory consumption, improving overall training efficiency.
[0099] In terms of implementation, the input image resolution was set to 256x256, and VQ-GAN was used to convert the image into a 16×16 semantic label sequence, totaling 256 labels. During training, a weak random cropping and resizing strategy was adopted to randomly scale the image size between 0.8 and 1. Furthermore, the model used a pre-trained basic Transformer architecture with ViT-B as the encoder. The AdamW optimizer was used for 960 epochs of training. The batch size for each epoch was 24, and a cosine learning rate decay strategy was employed, with an initial learning rate of 1.5×10⁻⁴ and a warm-up period of 80 stages. All experiments were performed on a V100 GPU to ensure high efficiency for large-scale training.
[0100] In summary:
[0101] The MCG model proposed in this invention effectively bridges the gap between visual and textual semantics by introducing a masking strategy with variable masking rates and a category label filling mechanism, thereby improving the accuracy and efficiency of image generation. This method significantly improves generation performance, training efficiency, and computational resource consumption, demonstrating strong application prospects and practical significance.
[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A variable mask based text synthesis image method, characterized in that: At least comprising the following steps: S1: build a multi-modal image generation model, namely MCG model, which converts image and text information into unified semantic labels through VQ-GAN and CLIP pre-training model, realizes effective fusion of visual and text information, and lays a foundation for subsequent processing; S2: information compression and reconstruction of MCG model, adopts variable masking rate strategy to randomly mask part of the semantic labels, forces the model to learn to reconstruct the complete content from part of the information, improves the understanding ability and generation efficiency of the model; S3: a learnable filling mechanism is adopted to further optimize the integrity of the input label sequence; S4: efficient feature processing, MCG model adopts ViT encoder-decoder structure, which can effectively process long sequence input, capture global semantic information, and improve the understanding and generation ability of the model to image content; S5: optimization training strategy, reconstruction training mechanism and contrast loss optimization work together to improve the semantic consistency and feature representation ability of the model, and reconstruction loss ensures that the model can accurately reconstruct the masked information, while contrast loss enhances the model's ability to distinguish different samples; S6: improve the computational efficiency, adopt the mask label deletion mechanism to reduce the number of labels that need to be processed, significantly reduce the computational complexity and memory consumption, and improve the training efficiency.
2. The variable mask based text synthesis image method of claim 1, wherein: The S1 at least comprises the following steps: Under the coding of VQ-GAN, the input image is first encoded into a set of discrete visual features, which reflect different semantic information in the image, that is, the semantic labels of the image are obtained; The CLIP pre-training model uses the pre-trained visual-language joint embedding space to map the corresponding text information to the same semantic space, so as to obtain the semantic representation of the text, that is, the semantic labels of the text are obtained; In this way, the semantic labels of images and texts are aligned in the same space, providing a good semantic basis for subsequent processing; Mix the semantic labels of the image and the semantic labels of the text; The mixed semantic label sequence will be used as the input of the model.
3. The variable mask based text synthesis image method of claim 2, wherein: The variable masking rate strategy is to randomly select a masking rate R from a truncated Gaussian distribution with 0.55 as the center, 0.5 as the left truncation, and 1 as the right truncation; The masking rate R determines the proportion of labels that will be randomly masked; The masked labels are replaced by a learnable "fake" label [M], which will be continuously optimized during training to help the model better fill the masked part.
4. The variable mask-based text synthesis image method of claim 3, wherein: The learnable filling mechanism is to fill the gap by introducing a learnable "fake" label [C0] in the masked label part.
5. The variable mask based text synthesis image method of claim 4, wherein: The MCG model uses a ViT encoder-decoder structure to process the edited label sequence; The ViT encoder takes the masked label sequence as input and maps it to the latent feature space, and the ViT encoder can not only extract information from local label features, but also summarize image features in a global range; The class label [C] of each image acts as a representative of visual features in this process, helping the MCG model to understand image content from a global perspective; The latent feature sequence obtained by the ViT encoder is further processed by the decoder; Before decoding, the MCG model first fills the ViT encoder output with class labels [C] to restore it to the same length as the original input; The decoder reconstructs based on these filled features, gradually restoring the complete information of the image and text semantic labels.
6. The variable mask-based text synthesis image method of claim 5, wherein: In the reconstruction training mechanism: Let represents the latent label sequence obtained from the input image and text annotation, where N is the length of the label sequence; Let denotes a binary mask mask used to determine which tags are masked; The training target of the MCG model is to reconstruct the masked label from the unmasked label, so the cross-entropy loss function is added between the actual one-hot label and the decoder output, specifically defined as: where Y M is the unmasked subset of the label sequence Y, p(y i | Y M ) is the label probability prediction under the encoder-decoder network conditioned on the unmasked labels Y M and is the expectation over all samples Y in the dataset ; The calculation formula of the contrastive loss is as follows: where z denotes the normalized features from two layers of MLP, B denotes the batch size, and T is the temperature parameter, augmented view from the same image, negative sample z j from other samples in the same batch; In this way, the contrastive loss is introduced to optimize the representation learning and improve the performance of the model in multi-modal information generation; The final training target loss function is the weighted sum of the reconstruction loss and the contrastive loss, represented as: Where λ = 0.1 is used to balance the proportion of the two losses.
7. The variable mask based text synthesis image method of claim 6, wherein: The deletion mechanism of the masked label is that for the masked label, not only is it masked, but also a part of it is randomly deleted.