A method, apparatus, device and medium for generating an image based on text

Abstract

This invention discloses a method, apparatus, device, and medium for generating images based on text. The method includes: acquiring a training dataset comprising multiple cross-modal data pairs; extracting semantic features from images in the cross-modal data pairs using an image coding tower to obtain a target image embedding vector; extracting semantic features from text prompts in the cross-modal data pairs using a text coding tower to obtain a reference text embedding vector; predicting image embedding vectors from the reference text embedding vectors using a cross-modal predictor to obtain a predicted image embedding vector; and jointly training the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector, thereby using the trained text coding tower and diffusion model to determine the generated image corresponding to the text prompt to be processed. This scheme introduces a joint embedding prediction mechanism to generate images corresponding to text, improving the semantic understanding of complex text and enhancing the consistency between the generated content and the text semantics.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating images based on text. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, it has been widely applied in fields such as urban governance, healthcare, and industry. Text-to-image (TTO) technology, as a core branch of multimodal content generation, is increasingly becoming a key tool for driving intelligent industrial upgrading. For example, in urban planning and management, TTO technology can quickly generate high-quality virtual cityscapes to showcase future urban planning schemes and help decision-makers intuitively understand complex urban spatial layouts. This application scenario places extremely high demands on the accuracy and reliability of the generated content, as well as a deep understanding of complex instructions.

[0003] In related technologies, a diffusion model-based generative approach is employed, such as StableDiffusion and Kolors. This approach trains the model on large-scale text-image pairs to achieve the conversion from text descriptions to images. These techniques typically rely on pre-trained models like CLIP (Contrastive Language-Image Pretraining) for text encoding, and then gradually generate images corresponding to the text through a diffusion process. CLIP, as a cross-modal pre-trained model, focuses on using contrastive learning to deeply associate images with natural language text, enabling the model to understand the semantic correspondence between image content and text descriptions.

[0004] However, the model training process of the above-mentioned schemes heavily relies on fitting the surface statistical regularities in massive amounts of data, rather than abstractly understanding the deep semantic relationships. Specifically, the core training goal of existing models is pixel-level reconstruction, that is, pursuing a high-fidelity match between generated pixels and training data pixels in detail. This leads the models to tend to learn surface, statistical texture and style features, rather than deeply understanding the complex semantic relationships and logical structures behind the text. Therefore, when processing texts that require complex semantic understanding and logical reasoning, the models often exhibit problems such as insufficient semantic alignment and poor rationality of generated content, failing to guarantee the quality and practicality of the generated content, thus limiting their application effectiveness in high-reliability scenarios such as urban governance. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for generating images based on text. It combines a text encoding tower trained by introducing a joint embedding prediction mechanism with a diffusion model to generate images corresponding to the text, thereby improving the model's semantic understanding of complex text, enhancing the consistency between the generated content and the text semantics, and improving the model's adaptability to different application scenarios.

[0006] According to one aspect of the present invention, a method for generating images based on text is provided, the method comprising: Obtain a training dataset comprising multiple cross-modal data pairs; wherein the cross-modal data pairs include images and their corresponding text prompts; The target image embedding vector is obtained by extracting semantic features from the images in the cross-modal data pair using an image coding tower, and the reference text embedding vector is obtained by extracting semantic features from the text prompts in the cross-modal data pair using a text coding tower. The reference text embedding vector is used to predict the image embedding vector by a cross-modal predictor to obtain the predicted image embedding vector; wherein, the cross-modal predictor is used to model the mapping relationship from the text embedding vector to the image embedding vector; The text coding tower and the cross-modal predictor are jointly trained based on the target image embedding vector and the predicted image embedding vector, so as to determine the generated image corresponding to the text prompt to be processed using the trained text coding tower and diffusion model; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

[0007] According to another aspect of the present invention, an apparatus for generating images based on text is provided, the apparatus comprising: The training data acquisition module is used to acquire a training dataset that includes multiple cross-modal data pairs; wherein, the cross-modal data pairs include images and their corresponding text prompts; The semantic feature extraction module is used to extract semantic features from the images in the cross-modal data pair using an image coding tower to obtain a target image embedding vector, and to extract semantic features from the text prompts in the cross-modal data pair using a text coding tower to obtain a reference text embedding vector. A cross-modal prediction module is used to predict the image embedding vector by performing image embedding vector prediction on the reference text embedding vector through a cross-modal predictor; wherein, the cross-modal predictor is used to model the mapping relationship from text embedding vector to image embedding vector; An image generation module is used to jointly train the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector, so as to determine the generated image corresponding to the text prompt to be processed using the trained text coding tower and diffusion model; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text-based image generation method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the text-based image generation method according to any embodiment of the present invention.

[0010] The technical solution of this invention involves acquiring a training dataset comprising multiple cross-modal data pairs. Each cross-modal data pair includes an image and its corresponding text prompt. A target image embedding vector is obtained by extracting semantic features from the image in the cross-modal data pair using an image encoding tower, and a reference text embedding vector is obtained by extracting semantic features from the text prompt using a text encoding tower. A predicted image embedding vector is obtained by predicting the image embedding vector from the reference text embedding vector using a cross-modal predictor. The cross-modal predictor models the mapping relationship between the text embedding vector and the image embedding vector. The text encoding tower and the cross-modal predictor are jointly trained based on the target image embedding vector and the predicted image embedding vector to determine the generated image corresponding to the text prompt using the trained text encoding tower and diffusion model. The diffusion model generates the image corresponding to the text prompt. This technical solution combines a text encoding tower trained through a joint embedding prediction mechanism with a diffusion model to generate images corresponding to text, improving the model's semantic understanding of complex text, enhancing the consistency between the generated content and the text semantics, and improving the model's adaptability to different application scenarios, thus ensuring the quality and practicality of the generated content.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for generating images based on text, according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a joint training phase according to an embodiment of the present invention; Figure 3 This is a flowchart of another method for generating images based on text, provided by an embodiment of the present invention; Figure 4 This is a flowchart illustrating a reasoning stage according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an image generation device based on text according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device that implements a text-based image generation method according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a method for generating images based on text, provided in Embodiment 1 of the present invention. This embodiment is applicable to generating images that have semantic consistency with the text. The method can be executed by a device for generating images based on text, which can be implemented in hardware and / or software and can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes: S110, Obtain a training dataset that includes multiple cross-modal data pairs; wherein, the cross-modal data pairs include images and their corresponding text prompts.

[0017] In this embodiment, a training dataset is first obtained. This training dataset includes multiple cross-modal data pairs, each of which includes an image and its corresponding text prompt. For example, the training dataset can be represented as follows: ,in, For the first Images in a cross-modal data pair For the first Text prompts in cross-modal data pairs, and and There is a corresponding relationship. The text prompts are described using natural language.

[0018] S120: Semantic features are extracted from the images in the cross-modal data pair using an image coding tower to obtain the target image embedding vector, and semantic features are extracted from the text prompts in the cross-modal data pair using a text coding tower to obtain the reference text embedding vector.

[0019] Image coding towers are used to map images into fixed-dimensional image embedding vectors by extracting semantic features. These image embedding vectors reflect the overall semantic content of the image, filtering out low-level details such as texture, noise, and lighting, while preserving high-level semantics such as object categories, spatial relationships, scene layout, and color distribution. For example, image coding towers can be built based on neural network models, such as VisionTransformer (ViT) or deep convolutional neural networks. VisionTransformer segments the image into smaller pieces and uses a Transformer to interpret the image. Deep convolutional neural networks can use ResNet-50 or EfficientNet. The target image embedding vector refers to the image embedding vector obtained by extracting semantic features from the image through the image coding tower.

[0020] The text encoding tower is used to map text prompts into fixed-dimensional text embedding vectors by extracting semantic features from the text prompts. These text embedding vectors capture high-level semantic information such as objects, attributes, spatial relationships, and actions within the text prompts. For example, the text encoding tower can employ a Transformer encoder, which is built upon a multi-layer self-attention and feedforward network. For instance, the Transformer encoder could be BERT or T5. The reference text embedding vector refers to the text embedding vector obtained through semantic feature extraction using the text encoding tower, and it is necessary to ensure that the reference text embedding vector has the same embedding dimension as the target image embedding vector.

[0021] In this embodiment, after obtaining a training dataset including multiple cross-modal data pairs, the images from the cross-modal data pairs can be input into the image coding tower. Through image coding tower Semantic features are extracted from the image, and the image is then mapped to a semantic embedding vector of the same dimension as the target image embedding vector. Specifically, this can be represented as follows: .in, Embed vectors for the target image. For any cross-modal data pair, the image, This is for the embedding dimension. Simultaneously, textual prompts from cross-modal data pairs can be input into the text encoding tower. Through text encoding tower Semantic features are extracted from the text prompt, and the text prompt is then mapped to a semantic embedding vector with the same embedding dimension as the image embedding vector, which serves as the reference text embedding vector. Specifically, this can be represented as follows: .in, For the reference text embedding vector, Text prompts corresponding to images in cross-modal data pairs.

[0022] S130, the reference text embedding vector is used to predict the image embedding vector by using a cross-modal predictor to obtain the predicted image embedding vector; wherein, the cross-modal predictor is used to model the mapping relationship from the text embedding vector to the image embedding vector.

[0023] In this embodiment, after obtaining the reference text embedding vector, it can be input into the cross-modal predictor. The predicted image embedding vector is obtained by using a pre-modeled mapping relationship to predict the image embedding vector from the reference text embedding vector. Specifically, it can be expressed as Among them, cross-modal predictors The training objective is to predict image embedding vectors. Approximate the target image embedding vector as closely as possible However, its low-level features cannot be directly copied; instead, its high-level semantic structure should be learned. Since the training images were already processed in step S120... The pixel mapping is a semantic abstraction, so the task of using a cross-modal predictor can be to understand the text and reconstruct this semantic summary, rather than memorizing the RGB values ​​of the image.

[0024] It's important to note that even if the input to a cross-modal predictor is in the form of semantic embeddings, if the predictor is too powerful or information-intensive, shortcut learning may still occur. Shortcut learning refers to the model prioritizing the learning of superficial, non-essential "shortcut features" in the data during training, rather than truly understanding the inherent semantic patterns of the data. While this "shortcut" can quickly reduce loss on the training set, it leads to a decline in the model's generalization ability and a collapse in performance on the test set or in real-world scenarios.

[0025] To address the aforementioned shortcut learning phenomenon, the cross-modal predictor can be structurally optimized by setting a lightweight cross-modal predictor that forces it to learn semantic features. Optionally, the cross-modal predictor can employ a multilayer perceptron with a target number of layers or a Transformer decoder with residual connections, where the Transformer decoder is constructed based on cross-attention. The target number of layers can be set according to actual needs, but it must meet the requirement of being lightweight; for example, the target number of layers can be set to 1-3 layers. For instance, the cross-modal predictor can be set as a multilayer perceptron with 1-3 layers or a small Transformer decoder with residual connections.

[0026] This scheme sets up the cross-modal predictor as a lightweight predictor with limited capacity, preventing it from memorizing complex low-level mappings and allowing it to learn only the most salient semantic rules. This guides the cross-modal predictor to focus on high-level structures (i.e., inductive bias), thus effectively avoiding shortcut learning.

[0027] Furthermore, the training images used in the training dataset of this invention are complete and unmasked, without any enhancement or perturbation. This limits the information pathway of the cross-modal predictor, prevents image cue leakage, eliminates the possibility of "copying," and forces the cross-modal predictor to learn semantic features. In addition, for the image coding tower pre-trained on large-scale data, its embedding space already possesses class discriminativeness, attribute decoupling (color, shape, and material can be separated), and spatial structure encoding capabilities, i.e. It is itself a carrier of high-level semantic structure. In this context, cross-modal predictors only need to learn how to obtain semantic vectors equivalent to image embeddings from text embeddings, rather than low-level features.

[0028] S140, the text coding tower and cross-modal predictor are jointly trained based on the target image embedding vector and the predicted image embedding vector, so as to use the trained text coding tower and diffusion model to determine the generated image corresponding to the text prompt to be processed; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

[0029] In this embodiment, after obtaining the target image embedding vector and the predicted image embedding vector, the text coding tower and the cross-modal predictor can be jointly trained based on them. This process is an end-to-end semantic reasoning process, rather than a simple feature transfer. Optionally, the joint training of the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector includes: performing a gradient truncation operation on the target image embedding vector to obtain a reference image embedding vector; determining a target loss function based on the distance between the reference image embedding vector and the predicted image embedding vector; and updating the training parameters of the text coding tower and the cross-modal predictor based on the target loss function.

[0030] Specifically, when jointly training the text coding tower and the cross-modal predictor, a gradient truncation operation needs to be performed on the target image embedding vector to obtain the reference image embedding vector. The gradient truncation operation (Stop-Gradient) means that the target image embedding vector is computed normally during forward propagation, but treated as a constant during backpropagation, without calculating its gradient or updating the image coding tower parameters. It's important to note that using the gradient truncation operation is mandatory; without it, the optimization objective becomes: At this point, the text coding tower and cross-modal predictor can easily map all inputs to a zero vector (or other constant), leading to representation collapse, where even if the embeddings are informationless, they cannot be used for generation. Introducing Stop-Gradient, the reference image embedding vector becomes a fixed reference point, forcing the text coding tower... and cross-modal predictors Learn how to proactively approximate a stable semantic representation to obtain meaningful cross-modal alignment.

[0031] After obtaining the reference image embedding vector, the target loss function can be determined based on the distance between the reference image embedding vector and the predicted image embedding vector. The core idea of ​​the target loss function is to minimize the distance between the predicted image embedding vector and the reference image embedding vector, while preventing representation collapse. Optionally, the target loss function can be expressed as follows: ;in, Let be the target loss function. Let be the distance function. This indicates the gradient cutoff operation. Indicates a cross-modal predictor. Represents the text encoding tower, Represents the image coding tower. For text prompts, The image corresponding to the text prompt. Embed vectors for the target image. Embedded vectors for reference images, For the reference text embedding vector, To predict the image embedding vector. For example, the distance function can be the mean squared error (MSE) or the cosine distance.

[0032] After determining the target loss function, the training parameters of the text coding tower and the cross-modal predictor can be updated based on the target loss function until the prediction accuracy requirements are met (this can be flexibly set according to the actual application scenario). It should be noted that, to prevent representation collapse, the image coding tower... It does not participate in the backpropagation process, only the text encoding tower. With cross-modal predictors It participates in the backpropagation process and updates training parameters through gradient calculation. Specifically, the text encoding tower... With cross-modal predictors This forms a unified, differentiable computation graph, which can be jointly updated using standard optimizers (such as AdamW), with the following update formula: ， ; in, Text encoding tower Trainable parameters, For cross-modal predictors Trainable parameters, The learning rate is used. For example, during backpropagation, a strategy that allows fine-tuning of all layers or a strategy that only fine-tunes the top layer can be employed. To improve generalization, regularization techniques such as dropout and gradient clipping can be introduced. Furthermore, since the cross-modal predictor is set as a lightweight predictor, its parameter count is much smaller than that of text coding towers and image coding towers.

[0033] Among them, based on actual needs, the image coding tower Two strategies can be adopted: 1. Use a pre-trained image coding tower with completely frozen parameters on large-scale data. Its parameters are not updated throughout the training process. This method is suitable for scenarios with limited training resources. 2. EMA (Exponential Moving Average) update: This method requires maintaining two image encoders, including an online encoder and a target encoder. The online encoder is a pre-trained encoder with frozen parameters. It does not participate in any gradient calculation or parameter update during the entire training process and is just a "parameter source". The parameters of the target encoder are slowly updated from the online encoder through EMA and directly participate in the loss calculation, outputting the target image embedding vector. However, due to the use of stop-gradient, it is not updated through backpropagation. That is, the target encoder is used as the image coding tower.

[0034] For example, the EMA update process can be represented as follows: ;in, For the learnable parameters of the target encoder, For parameters of the online encoder, This is the momentum coefficient (which can be used to control the update rate). When the target encoder is initialized randomly, it can provide a smooth transition in the early stages of training, thus avoiding abrupt changes in the output of the image coding tower.

[0035] Figure 2A flowchart of a joint training phase is shown, in which the training images are the images in the cross-modal data pair, the image embedding z_i is the target image embedding vector, the text embedding z_t is the reference text embedding vector, the predicted image embedding p_i is the predicted image embedding vector, and the JEPA loss function is the target loss function. This invention introduces a joint embedding prediction mechanism in the joint training phase. This phase mainly includes an image encoding tower, a text encoding tower, and a cross-modal predictor. The specific process is as follows: 1. Input the training image into the image encoding tower (i.e., E_I in the figure) to extract image semantic features and obtain image embedding z_i. Input the corresponding text prompt into the text encoding tower (i.e., E_T in the figure) to extract text semantic features and obtain text embedding z_t; 2. Input the text embedding z_t into the cross-modal predictor P to predict the image embedding vector and obtain the predicted image embedding p_i; 3. Perform Stop-Gradient on the image embedding z_i and calculate the JPEG loss function in combination with the predicted image embedding p_i; 4. Backpropagate the JEPA loss function to the text encoding tower (i.e., E_T in the figure) and the cross-modal predictor P for joint training. When the prediction accuracy requirement is met, the joint training process stops, and the joint training is completed. Among them, JEPA (Joint Embedding Predictive Architecture) is a self-supervised learning framework that aims to efficiently learn the semantic laws of the physical world by making predictions in an abstract representation space rather than generating raw data.

[0036] This invention upgrades the conditional learning of text-based graphs from "fuzzy matching" to "structured semantic reconstruction" by "supervised reconstruction in semantic space," thereby naturally avoiding the problem of low-level feature replication and allowing focus on high-level semantic alignment.

[0037] After joint training, the trained text encoding tower can be used as the text encoder of the diffusion model, and combined with the diffusion model to determine the generated image corresponding to the text prompt to be processed. The text prompt to be processed can refer to the text prompt awaiting text-to-image processing, which can be input by the user through a preset data interface; the generated image can refer to the image corresponding to the text prompt to be processed, generated using the text encoding tower and the diffusion model. In the inference stage of the text-to-image processing, this invention mainly includes a parameter-frozen text encoding tower and a pre-trained diffusion model. The parameter-frozen text encoding tower is the text encoding tower obtained through joint training, and its parameters are fixed after training. It should be noted that the image encoding tower and cross-modal predictor do not participate in the calculation during the generation stage; they are only used during the joint training stage to enhance the semantic alignment capability of the text encoding tower.

[0038] The diffusion model, used to generate images corresponding to text prompts, includes a text encoder, a denoising network, a diffusion scheduler, and a decoder. Specifically, the text encoder encodes the text prompts, transforming natural language into a machine-understandable digital representation. The denoising network estimates noise in the latent space representation at each time step based on the encoded information. The diffusion scheduler updates the latent space representation based on the noise estimation results. The decoder decodes the latent space representation back to the image space representation. The latent space refers to the compressed space of the data; processing data within the latent space significantly reduces computational load, thus improving processing efficiency. The core of the diffusion model lies in iteratively denoising the initially generated random noise latent space representation under the guidance of the text prompts, in order to reconstruct a high-quality image corresponding to the text prompts from the initial random noise latent space representation.

[0039] It should be noted that, compared to traditional CLIP-based text-to-image (TPI) methods, the advantages of this invention are: 1. A cross-modal predictor is introduced, which, through the training objective of "predicting image semantic embeddings from text embeddings," forces the text representation to contain information sufficient to reconstruct the high-level semantic structure of the image (such as object attributes, spatial relationships, logical interactions, etc.). Compared to CLIP, which only optimizes image-text similarity, the embeddings learned in this invention are optimized for "reconstructing image semantics," naturally containing the structured semantic information required for image generation tasks. As a conditional signal for the diffusion model, it is more expressive and has greater control precision, which can improve the performance of the generated results in terms of semantic consistency and logical rationality. 2. The prediction function learned in this invention is decomposable and composable, which can effectively decouple conceptual elements (such as subject, attribute, style, and relationship). Thus, when faced with novel text combinations that did not appear in the training, it can still maintain the correct binding of attributes and subjects, reducing the risk of semantic drift or logical mismatch.

[0040] 3. This invention adopts a positive sample-driven predictive learning paradigm and introduces a joint embedding prediction architecture to align semantic features between different modalities. It achieves feature matching of different modalities in the "embedding space" without relying on large-scale negative sample sampling. The training effect does not fluctuate drastically with the batch size and can be trained stably and efficiently with a small batch size. It can significantly reduce the requirements for hardware resources and improve reproducibility and deployment flexibility.

[0041] The technical solution of this invention involves acquiring a training dataset comprising multiple cross-modal data pairs. Each cross-modal data pair includes an image and its corresponding text prompt. A target image embedding vector is obtained by extracting semantic features from the image in the cross-modal data pair using an image encoding tower, and a reference text embedding vector is obtained by extracting semantic features from the text prompt using a text encoding tower. A predicted image embedding vector is obtained by predicting the image embedding vector from the reference text embedding vector using a cross-modal predictor. The cross-modal predictor models the mapping relationship between the text embedding vector and the image embedding vector. The text encoding tower and the cross-modal predictor are jointly trained based on the target image embedding vector and the predicted image embedding vector to determine the generated image corresponding to the text prompt using the trained text encoding tower and diffusion model. The diffusion model generates the image corresponding to the text prompt. This technical solution combines a text encoding tower trained through a joint embedding prediction mechanism with a diffusion model to generate images corresponding to text, improving the model's semantic understanding of complex text, enhancing the consistency between the generated content and the text semantics, and improving the model's adaptability to different application scenarios, thus ensuring the quality and practicality of the generated content.

[0042] Example 2 Figure 3 This is a flowchart of a method for generating images based on text, provided in Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, the optimization is as follows: determining the generated image corresponding to the text prompt to be processed using a trained text encoding tower and a diffusion model, including: encoding the text prompt to be processed using the trained text encoding tower to obtain a target text embedding vector; and using the diffusion model to perform latent space iterative denoising on the initial random noise based on the target text embedding vector to obtain the generated image corresponding to the text prompt to be processed.

[0043] like Figure 3 As shown, the method in this embodiment specifically includes the following steps: S210, Obtain a training dataset that includes multiple cross-modal data pairs; wherein, the cross-modal data pairs include images and their corresponding text prompts.

[0044] S220: Semantic features are extracted from the images in the cross-modal data pair using an image coding tower to obtain the target image embedding vector, and semantic features are extracted from the text prompts in the cross-modal data pair using a text coding tower to obtain the reference text embedding vector.

[0045] S230, semantic features are extracted from the images in the cross-modal data pair using an image coding tower to obtain the target image embedding vector, and semantic features are extracted from the text prompts in the cross-modal data pair using a text coding tower to obtain the reference text embedding vector.

[0046] S240, jointly trains the text coding tower and cross-modal predictor based on the target image embedding vector and the predicted image embedding vector.

[0047] The specific implementation of S210-S240 can be referred to the relevant description in the above embodiments, and will not be repeated here.

[0048] S250 uses the trained text encoding tower to encode the text prompts to be processed to obtain the target text embedding vector.

[0049] In this embodiment, after joint training is completed, the user-input text prompts can be encoded using the trained text encoding tower. The text prompts are then transformed into fixed-dimensional semantic embedding vectors as target text embedding vectors. These target text embedding vectors are used as the semantic condition source of the diffusion model, which can be used to optimize the predictability of text embeddings to the semantic structure of images.

[0050] S260, by using a diffusion model to perform latent space iterative denoising on the initial random noise based on the target text embedding vector, the generated image corresponding to the text prompt to be processed is obtained.

[0051] After obtaining the target text embedding vector, a latent space representation of random noise can be initialized using a diffusion model. Guided by the target text embedding vector, the latent space representation of the initial random noise is then denoised step-by-step, ultimately outputting the generated image corresponding to the text prompt to be processed. Optionally, the generated image corresponding to the text prompt to be processed is obtained by performing iterative latent space denoising on the initial random noise based on the target text embedding vector using a diffusion model. This includes: compressing the initial random noise using a diffusion model to obtain an initial latent space representation; estimating the noise in the initial latent space representation based on the target text embedding vector and the initial time step to obtain an initial noise estimation result; where the initial time step is an integer greater than 1; determining the target latent space representation corresponding to the next time step based on the initial latent space representation and the initial noise estimation result; where the next time step is equal to the difference between the initial time step and 1; using the next time step as the initial time step and the target latent space representation as the initial latent space representation, the above noise estimation process and target latent space representation determination process are repeated until the next time step is less than 1; and decompressing the last determined target latent space representation to obtain the generated image corresponding to the text prompt to be processed.

[0052] Specifically, the initial random noise is first input into the diffusion model, and the diffusion model compresses the initial random noise into a pre-defined latent space to obtain the initial latent space representation. Then embed the target text into the vector. As a conditional signal injected into the denoising network of the diffusion model, text-image alignment is achieved through the cross-attention mechanism in the denoising network, and the target text embedding vector is then embedded. Guided by the initial time step Corresponding initial latent space representation Perform noise estimation and finally output the initial noise estimation result. The cross-attention mechanism can be represented as follows: ,in, These represent query, key, and value, respectively. Latent space features extracted from a denoising network, and From the target text embedding vector , This indicates transpose. For example, the denoising network can be a U-Net network.

[0053] Then, the diffusion scheduler is used based on the initial latent space representation. and initial noise estimation results Determine the next time step Corresponding target latent space representation For example, the target latent space is represented according to the formula... To be updated, among which, Indicates time step ( The corresponding latent space representation, Indicates time step The corresponding latent space representation, Indicates the single-step signal retention rate (i.e., time step). (signal retention rate) Represents the cumulative signal retention rate (i.e., from time step). Time to step (overall signal retention rate) This represents the posterior variance (used to describe how much noise was added back). This indicates random noise sampling (the diffusion model needs to resample at each time step). express The corresponding noise estimation results. It should be noted that the target text embedding vector... Throughout the iterative denoising process, the latent space representation remains constant, meaning that the same target text embedding vector is used to update the latent space representation at each time step. Furthermore, the number of time steps can be set according to actual needs; for example, the number of time steps can be set to 1000.

[0054] Based on the above update formula, using and The time step can be calculated. Corresponding target latent space representation Next, let's... Denoising network is used to process time steps Corresponding target latent space representation Perform noise estimation and output time step. Corresponding noise estimation results reuse and Calculate the time step Corresponding target latent space representation Referring to the above process for noise estimation and target latent space representation determination, let the time step... Decrease sequentially to obtain the time step. Corresponding target latent space representation until the time step is The iterative denoising process stops at this point, and the final determined latent space representation of the target can be obtained. , It can be considered as a noise-free latent space representation of the image. Finally, it can be decoded... The data is decompressed and restored to its image space representation, thereby obtaining the generated image corresponding to the text prompt to be processed.

[0055] Figure 4 A flowchart illustrating the inference phase is presented, where the user text prompt is the text prompt to be processed, the frozen text encoding tower E_I is the trained text encoding tower, the text embedding z_t1 is the target text embedding vector, Cross-Attention represents cross-attention, and the denoised image is the target latent space representation. Figure 4 As shown, the reasoning process of the generated image is as follows: 1. Text encoding: The user's text prompt is input into the frozen text encoding tower E_I for encoding to obtain the text embedding z_t1; 2. Iterative denoising: For each time step t, the latent space representation corresponding to random noise is generated through the diffusion model. The text embedding z_t1 is injected into the denoising network as a conditional signal. Under the guidance of the text embedding z_t1, the denoising network generates the noise estimation result corresponding to the target latent space representation at time step t. Then, the diffusion scheduler generates the target latent space representation at time step t-1 based on the target latent space representation at time step t and its noise estimation result. This process is repeated for T steps to obtain the final determined target latent space representation; 3. Data decoding: The final determined target latent space representation is decoded by the decoder to restore it to the image space representation, thereby obtaining the final generated image.

[0056] The technical solution of this invention, when determining the generated image corresponding to the text prompt to be processed using a trained text encoding tower and a diffusion model, encodes the text prompt to obtain a target text embedding vector using the trained text encoding tower; then, the diffusion model performs latent space iterative denoising on the initial random noise based on the target text embedding vector to obtain the generated image corresponding to the text prompt. This technical solution combines a text encoding tower trained by introducing a joint embedding prediction mechanism with a diffusion model to generate the image corresponding to the text, improving the model's semantic understanding ability of complex text, enhancing the consistency between the generated content and the text semantics, and improving the model's adaptability to different application scenarios, ensuring the quality and practicality of the generated content.

[0057] Example 3 Figure 5 This is a schematic diagram of a device for generating images based on text, provided in Embodiment 3 of the present invention. This device can execute the method for generating images based on text provided in any embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 5 As shown, the device includes: The training data acquisition module 310 is used to acquire a training dataset including multiple cross-modal data pairs; wherein, the cross-modal data pairs include images and their corresponding text prompts; The semantic feature extraction module 320 is used to extract semantic features from the images in the cross-modal data pair through an image coding tower to obtain a target image embedding vector, and to extract semantic features from the text prompts in the cross-modal data pair through a text coding tower to obtain a reference text embedding vector. The cross-modal prediction module 330 is used to predict the image embedding vector by performing image embedding vector prediction on the reference text embedding vector through a cross-modal predictor; wherein, the cross-modal predictor is used to model the mapping relationship from text embedding vector to image embedding vector; The image generation module 340 is used to jointly train the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector, so as to determine the generated image corresponding to the text prompt to be processed using the trained text coding tower and diffusion model; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

[0058] Optionally, the image generation module 340 is used for: Gradient truncation is performed on the target image embedding vector to obtain the reference image embedding vector; The target loss function is determined based on the distance between the reference image embedding vector and the predicted image embedding vector; The training parameters of the text coding tower and the cross-modal predictor are updated based on the target loss function.

[0059] Optionally, the target loss function is expressed as follows: ; in, Let be the target loss function. Let be the distance function. This indicates the gradient cutoff operation. Indicates a cross-modal predictor. Represents the text encoding tower, Represents the image coding tower. For text prompts, The image corresponding to the text prompt. Embed vectors for the target image. Embedded vectors for reference images, For the reference text embedding vector, To predict the image embedding vector.

[0060] Optionally, the image generation module 340 is further configured to: The target text embedding vector is obtained by encoding the text prompt to be processed using the trained text encoding tower. By using a diffusion model to perform latent space iterative denoising on the initial random noise based on the target text embedding vector, the generated image corresponding to the text prompt to be processed is obtained.

[0061] Optionally, the image generation module 340 is further configured to: The initial latent space representation is obtained by compressing the initial random noise using a diffusion model. Based on the target text embedding vector and the initial time step, noise estimation is performed on the initial latent space representation to obtain the initial noise estimation result; wherein, the initial time step is an integer greater than 1; The target latent space representation corresponding to the next time step is determined based on the initial latent space representation and the initial noise estimation result; wherein, the next time step is equal to the difference between the initial time step and 1; Using the next time step as the initial time step and the target latent space representation as the initial latent space representation, the above noise estimation process and target latent space representation determination process are re-executed until the next time step is less than 1; The target latent space representation, which was determined last time, is decompressed to obtain the generated image corresponding to the text prompt to be processed.

[0062] Optionally, the image encoding tower uses a Vision Transformer or a deep convolutional neural network; the text encoding tower uses a Transformer encoder, which is constructed based on a multi-layer self-attention and feedforward network.

[0063] Optionally, the cross-modal predictor employs a multilayer perceptron with a target number of layers or a Transformer decoder with residual connections, the Transformer decoder being constructed based on cross-attention.

[0064] The device for generating images based on text provided in this embodiment of the invention can execute the method for generating images based on text provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0065] Example 4 Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0066] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0067] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0068] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for generating images based on text.

[0069] In some embodiments, the method for generating images based on text can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for generating images based on text described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for generating images based on text by any other suitable means (e.g., by means of firmware).

[0070] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0071] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0072] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0073] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0074] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0075] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0076] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0077] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating images based on text, characterized in that, The method includes: Obtain a training dataset comprising multiple cross-modal data pairs; wherein the cross-modal data pairs include images and their corresponding text prompts; The target image embedding vector is obtained by extracting semantic features from the images in the cross-modal data pair using an image coding tower, and the reference text embedding vector is obtained by extracting semantic features from the text prompts in the cross-modal data pair using a text coding tower. The reference text embedding vector is used to predict the image embedding vector by a cross-modal predictor to obtain the predicted image embedding vector; wherein, the cross-modal predictor is used to model the mapping relationship from the text embedding vector to the image embedding vector; The text coding tower and the cross-modal predictor are jointly trained based on the target image embedding vector and the predicted image embedding vector, so as to determine the generated image corresponding to the text prompt to be processed using the trained text coding tower and diffusion model; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

2. The method according to claim 1, characterized in that, Joint training of the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector includes: Gradient truncation is performed on the target image embedding vector to obtain the reference image embedding vector; The target loss function is determined based on the distance between the reference image embedding vector and the predicted image embedding vector; The training parameters of the text coding tower and the cross-modal predictor are updated based on the target loss function.

3. The method according to claim 2, characterized in that, The target loss function is expressed as follows: ； in, Let be the target loss function. Let be the distance function. This indicates the gradient cutoff operation. Indicates a cross-modal predictor. Represents the text encoding tower, Represents the image coding tower. For text prompts, The image corresponding to the text prompt. Embed vectors for the target image. Embedded vectors for reference images, For the reference text embedding vector, To predict the image embedding vector.

4. The method according to any one of claims 1-3, characterized in that, The generated image corresponding to the text prompt to be processed is determined using the trained text encoding tower and diffusion model, including: The target text embedding vector is obtained by encoding the text prompt to be processed using the trained text encoding tower. By using a diffusion model to perform latent space iterative denoising on the initial random noise based on the target text embedding vector, the generated image corresponding to the text prompt to be processed is obtained.

5. The method according to claim 4, characterized in that, By using a diffusion model to perform latent space iterative denoising on the initial random noise based on the target text embedding vector, a generated image corresponding to the text prompt to be processed is obtained, including: The initial latent space representation is obtained by compressing the initial random noise using a diffusion model. Based on the target text embedding vector and the initial time step, noise estimation is performed on the initial latent space representation to obtain the initial noise estimation result; wherein, the initial time step is an integer greater than 1; The target latent space representation corresponding to the next time step is determined based on the initial latent space representation and the initial noise estimation result; wherein, the next time step is equal to the difference between the initial time step and 1; Using the next time step as the initial time step and the target latent space representation as the initial latent space representation, the above noise estimation process and target latent space representation determination process are re-executed until the next time step is less than 1; The target latent space representation, which was determined last time, is decompressed to obtain the generated image corresponding to the text prompt to be processed.

6. The method according to any one of claims 1-3, characterized in that, The image encoding tower uses a VisionTransformer or a deep convolutional neural network; the text encoding tower uses a Transformer encoder, which is constructed based on a multi-layer self-attention and feedforward network.

7. The method according to any one of claims 1-3, characterized in that, The cross-modal predictor employs a multilayer perceptron with a target number of layers or a Transformer decoder with residual connections, the Transformer decoder being constructed based on cross-attention.

8. An apparatus for generating images based on text, characterized in that, The device includes: The training data acquisition module is used to acquire a training dataset that includes multiple cross-modal data pairs; wherein, the cross-modal data pairs include images and their corresponding text prompts; The semantic feature extraction module is used to extract semantic features from the images in the cross-modal data pair using an image coding tower to obtain a target image embedding vector, and to extract semantic features from the text prompts in the cross-modal data pair using a text coding tower to obtain a reference text embedding vector. A cross-modal prediction module is used to predict the image embedding vector by performing image embedding vector prediction on the reference text embedding vector through a cross-modal predictor; wherein, the cross-modal predictor is used to model the mapping relationship from text embedding vector to image embedding vector; An image generation module is used to jointly train the text coding tower and the cross-modal predictor based on the target image embedding vector and the predicted image embedding vector, so as to determine the generated image corresponding to the text prompt to be processed using the trained text coding tower and diffusion model; wherein, the diffusion model is used to generate the image corresponding to the text prompt.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the method for generating images based on text as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for generating images based on text, as described in any one of claims 1-7.

Images