Training-free text-image generation method based on diffusion model

Abstract

The invention provides a training-free text-image generation method based on a diffusion model, and relates to the technical field of computer graphic processing and artificial intelligence. The method comprises the following steps: extracting semantic phrases and layout information in an input text by utilizing a natural language model, inputting the input text, the semantic phrases and the layout information as additional conditions into a diffusion model, and extracting cross attention maps of different time steps; a positive and negative sample concept and a foreground and background concept based on an object are constructed, a new loss function is calculated on a cross attention map for semantic information and layout information, the loss function combines semantic loss, regional loss and original loss of a diffusion model and is used for updating a potential space image, and the image is generated through iterative denoising and a decoder. According to the method, additional training is not needed, image generation output based on the diffusion model better meets text requirements, and a better text and image alignment effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer graphics processing and artificial intelligence, and relates to text semantic information processing and picture generation technology, in particular to a picture generation method based on a diffusion model. BACKGROUND

[0002] In recent years, breakthroughs in artificial intelligence and deep learning technology have revolutionized the field of cross-modal content synthesis through generative models. As one of the core research directions in this field, text-to-image (T2I) generation technology aims to synthesize high-quality images that match the semantics of natural language descriptions. This technology not only serves as a key path to achieving general artificial intelligence (AGI), but also has significant application value in digital art creation, virtual scene construction, intelligent advertisement generation, and educational visualization.

[0003] Early text-to-image (T2I) generation techniques were mainly based on generative adversarial networks (GANs) and variational autoencoders (VAEs). GANs achieve image generation through the dynamic game between discriminators and generators, and their generated images have made breakthroughs in terms of realism. However, they have inherent defects such as mode collapse and training oscillation. VAEs achieve data distribution fitting based on variational inference of latent variables, and have the advantage of theoretical rigor. However, the resolution of generated images is generally limited, and the problem of blurred details and textures is prominent. Since 2019, a new generation of generation framework based on diffusion probabilistic models (Diffusion Probabilistic Models) has rapidly become the mainstream paradigm in the field of image generation due to its stable training process, excellent generation quality, and strong multi-modal expansion capability.

[0004] Diffusion models have recently achieved unparalleled performance in many image synthesis tasks and generation tasks for other modalities such as video, audio, and text. Implicit diffusion models, which greatly reduce the computational cost of diffusion models, are widely used in various conditional generation tasks, including image editing and text-to-image generation. Among them, T2I diffusion models such as Stablediffusion, Imagen, and DALL-E 2 / 3 have shown strong capabilities in generating images with high aesthetic and realistic qualities.

[0005] In existing text-image generation techniques, the mainstream solution adopts an architecture combining diffusion probability models with attention mechanisms. This solution first converts the input text prompt into a conditional vector using an encoder such as CLIP, and then dynamically calculates the correlation between text and visual features through cross-attention mechanisms during the iterative denoising process of the diffusion model. This solution enhances the guidance of text features in the diffusion process through attention mechanisms, aiming to improve the consistency between the generated image and the input text.

[0006] In high-quality image generation tasks, semantic and texture consistency is a key factor in evaluating the quality of the generated results. Existing methods often overlook the regional characteristics and semantic differences during the generation process, resulting in discontinuous or ambiguous semantic expression in the generated images, especially at the junction of cross-scale features and foreground and background. On the one hand, traditional multi-scale feature fusion methods lack precise modeling of the correlation and significance distribution between different scales, and cannot effectively control the spatial position of different semantic elements in the image according to the text description, or dynamically adjust the generation strategy of different regions to strengthen certain details. For example, when describing a sentence, existing models can generate images containing elements, but may not accurately control the relative positions between these elements and the clarity of semantic features. On the other hand, the generation control precision is not ideal, and it is often not accurate when dealing with object positions, attributes, and other details, especially in multi-object scenes, it is difficult to achieve accurate layout control and effectively ensure the consistency between the generated content and the input text description. SUMMARY

[0007] To overcome the shortcomings of the prior art, the present application provides a training-free text-image generation method based on a diffusion model, comprising the following steps:

[0008] Obtain the input text, and use a pre-trained natural language model to extract semantic phrases and layout information from the input text;

[0009] Input the input text, semantic phrases, and layout information into the pre-trained diffusion model Stable Diffusion as additional conditions to generate a latent image, and extract cross-attention maps of the input text at different time steps;

[0010] Based on the cross-attention maps, construct a total loss function for the diffusion model Stable Diffusion, iteratively update the latent image based on the total loss function, remove the noise in the latent image, and obtain a generated image containing synthetic objects.

[0011] Further, the pre-trained natural language model is used to extract semantic phrases and layout information from the input text, and the specific method is as follows:

[0012] An input text with a word count of is obtained The input text is input into a pre-trained natural language model, and the understanding ability of the natural language model is used to perform fine-grained division on the input text , extract a plurality of semantic phrases and layout information contained in the input text, and obtain a semantic phrase set and a layout information set; The semantic phrase includes an entity noun and a modifier of the entity noun, and the layout information includes coordinate information of each entity noun.

[0013] The natural language model is used to extract semantic phrases in the input text

[0014] , and the extracted semantic phrase set is represented as . }={ }, wherein represents the position information of each semantic phrase in the input text , represents the entity noun in the th semantic phrase, represents the modifier of the entity noun in the th semantic phrase, represents and have a dependency relationship; set the generated picture size, use the natural language model to obtain the layout information of the entity noun, and the layout information is the position coordinates of the entity name in the generated picture. The layout information is converted into a mask set by binary mask conversion .

[0015] Further, the diffusion model Stable Diffusion includes a variational autoencoder VAE, a CLIP text encoder, and a U-Net noise predictor.

[0016] The input text, semantic phrase and layout information are input into the pre-trained diffusion model Stable Diffusion as additional conditions to generate a latent image, and the cross-attention graph of the input text at different time steps is extracted. The specific method is as follows:

[0017] The variational autoencoder VAE is used to sample a noise image from a standard normal distribution , and the noise image is compressed into a latent space to obtain a latent image .

[0018] The input text is preprocessed using the CLIP text encoder to obtain text labels .

[0019] ​The potential image and text label are input into the U-Net noise predictor, and the cross-attention map at different time steps is extracted, specifically as follows:

[0020] For the potential image , the denoiser is used to iteratively denoise to obtain the intermediate feature , is the latent space noise variable at time step t;

[0021] At different time steps, based on the text label with words and the intermediate feature , the learning matrix , of each word is projected to obtain the query matrix and the key matrix of each word, and the cross-attention map of each word is calculated to obtain cross-attention maps of words.

[0022] Further, the total loss function of the diffusion model Stable Diffusion is constructed based on the cross-attention map, including:

[0023] The object-based positive loss function and negative loss function are constructed to obtain the semantic loss function;

[0024] The object-based foreground loss function and background loss function are constructed, and the region loss function is constructed based on the foreground loss function and the background loss function;

[0025] The total loss function is established based on the semantic total loss function and the region loss function.

[0026] Further, the object-based positive loss function and negative loss function are constructed to obtain the semantic loss function, specifically as follows:

[0027] The positive loss function is established by minimizing the distance between all entity nouns and the corresponding modifiers, as shown in the following formula:

[0028]

[0029] wherein is the cross-attention map of the th word, is the cross-attention map of the th word, is the similarity between the cross-attention map and , as shown in the following formula:

[0030]

[0031] wherein, is KL divergence, used to calculate distance measure between cross attention maps;

[0032] The modifier and entity noun are compared with the rest of the words in the input text , and a negative loss function is established, as shown in the following formula:

[0033]

[0034] wherein, is a set of mismatched words obtained by excluding words irrelevant to the semantic phrase from the words in the input text , is the attention map corresponding to the given irrelevant word pair;

[0035] The positive loss function and the negative loss function are added to obtain the semantic loss function .

[0036] Further, the foreground loss function and the background loss function based on objects are constructed, and the region loss function is constructed based on the foreground loss function and the background loss function, and the specific method is:

[0037] Based on the cross attention map and the mask set, the foreground loss function is constructed, which is used to ensure that the synthesized object is contained in the bounding box, as shown in the following formula:

[0038]

[0039] wherein, is the element with the highest response selected;

[0040] The background loss function is constructed, which is used to prevent the synthesized object in the generated image from being generated outside the target area, as shown in the following formula:

[0041]

[0042] For all bounding boxes, the region loss function is constructed based on the foreground loss function and the background loss function, as shown in the following formula:

[0043]

[0044] wherein, is the th bounding box.

[0045] Further, the total loss function is established based on the semantic total loss function and the region loss function, and the specific method is:

[0046] The semantic loss, the region loss and the original loss of the diffusion model are combined to obtain the total loss function, as shown in the following formula:

[0047]

[0048] Among them, is a weight hyperparameter.

[0049] Further, the latent image is iteratively updated based on the total loss function to remove noise in the latent image, and a generated image containing a synthesized object is obtained, and the specific method is:

[0050] The latent image is updated based on the total loss function to obtain an updated latent image , as shown in the following formula:

[0051]

[0052] Among them, is the learning rate of time step t, which is usually linearly decayed with time step, and the update amplitude decreases as the denoising process proceeds, is the gradient operator of , indicating the partial derivative of the latent image ;

[0053] Noise in the latent image is removed by iterative updating to obtain a denoised latent image .

[0054] The denoised latent image is transmitted to the decoder to generate a generated image corresponding to the input text .

[0055] The beneficial effects produced by the above technical solutions are that: the training-free text-image generation method based on the diffusion model provided by the present application adopts a training-free mode, and the pre-trained models used are all in a frozen state, without the need for additional training; the present application modulates the feature vector using the prior knowledge space of the multi-modal model, thereby solving the attribute binding problem and the spatial layout problem of text-image generation, and improving the image-text matching degree and accuracy of the diffusion model generation result; the present application makes the image generation model based on the diffusion model output more in line with the requirements of the text, and can achieve better text and image alignment effect. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1Flowchart of the training-free text-image generation method based on diffusion model provided in Embodiment 1 of the present invention;

[0057] Figure 2 A semantic relevance diagram provided in Embodiment 1 of the present invention;

[0058] Figure 3 The present invention provides schematic diagrams of text-based image generation results based on different methods in Embodiment 1, wherein (a) is a schematic diagram of text-based image generation results based on the stable diffusion model, (b) is a schematic diagram of text-based image generation results based on the StructureDiffusion model, (c) is a schematic diagram of text-based image generation results based on the Attend and Excite model, and (d) is a schematic diagram of text-based image generation results based on the present invention. Detailed Implementation

[0059] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0060] Example 1

[0061] A training-free text-image generation method based on a diffusion model, such as Figure 1 As shown, it includes the following steps:

[0062] Step 1: Obtain the input text and use a pre-trained natural language model to extract semantic phrases and layout information from the input text;

[0063] The number of words obtained is Input text , input text The input is fed into a pre-trained natural language model, which then leverages the model's understanding capabilities to interpret the input text. Perform fine-grained segmentation to extract the input text. The semantic phrases and layout information contained therein are used to obtain a set of semantic phrases and a set of layout information.

[0064] The semantic phrase includes an entity noun and its modifiers, and the layout information includes the coordinates of each entity noun.

[0065] Extracting input text using natural language models In The extracted semantic phrase set is represented as { }={ },in,{ } indicates that each semantic phrase is in the input text Location information, Indicates the first entity nouns in a semantic phrase, representing the modifier of the entity noun in the semantic phrase, representing and there is a dependency relationship; set the generated picture size, use the natural language model to obtain the layout information of the entity noun, the layout information is the position coordinates of the entity name in the generated picture, and the layout information is binary mask converted to obtain a mask set .

[0066] For example, the prompt word text is "The soft towel was on top of the hard counter", which is input into the natural language model to obtain a set containing word position information [2, 3] (corresponding to'soft towel') and [9, 10] (corresponding to 'hard counter'); set the generated picture size to 512 * 512, the coordinates of the upper left corner to [0, 0], and the coordinates of the lower right corner to [512, 512], the bounding box should not overlap or exceed the image boundary, and the coordinate information of the entity noun obtained by the natural language model is layout information [[180, 120, 332, 220], [50, 260, 462, 480]]; for the direction information such as "on top of", the natural language model is used to identify and design a reasonable layout, and the layout information is binary mask converted to obtain a mask set .

[0067] Step 2: input the input text, semantic phrase and layout information as additional conditions into the pre-trained diffusion model Stable Diffusion, and extract the cross-attention graph of the input text at different time steps;

[0068] The diffusion model Stable Diffusion includes: a variational autoencoder VAE, a CLIP text encoder and a U-Net noise predictor;

[0069] The variational autoencoder VAE is used to sample a noise image from a standard normal distribution , and the noise image is compressed to a latent space to obtain a latent image ;

[0070] The CLIP text encoder is used to preprocess the input text to obtain text labels ;

[0071] The latent image and the text label are input into the U-Net noise predictor to extract the cross-attention graph at different time steps, and the specific method is:​

[0072] For the latent image , utilize the denoiser to iteratively denoise step by step to obtain the intermediate feature , is the latent space noise variable at time step t;

[0073] The loss function of the denoiser is shown in the following formula:

[0074]

[0075] The conditional action mechanism of the diffusion model Stable Diffusion can explicitly form the cross-attention between the text token and the denoiser intermediate feature .

[0076] At different time steps, based on the text token with words and the intermediate feature , the query matrix , and the key matrix of each word are obtained by projection, and the cross-attention map of each word is calculated to obtain the cross-attention map of words, and the cross-attention map of each word is shown in the following formula:

[0077]

[0078] wherein, is the feature dimension of the query matrix and the key matrix ;

[0079] Step 3: Construct object-based positive loss function and negative loss function to obtain semantic loss function;

[0080] By minimizing the distance between all entity nouns and corresponding modifiers, the positive loss function is established, as shown in the following formula:

[0081]

[0082] wherein, is the cross-attention map of the word, is the cross-attention map of the word, is the cross-attention map and​​​ similarity between them, as shown in the following formula:

[0083]

[0084] wherein, is the KL divergence, used to calculate the distance measure between cross attention maps;

[0085] The adjectives and entity nouns are compared with the rest of the words in the input text , and a negative loss function is established, as shown in the following formula:

[0086]

[0087] wherein, is the set of mismatched words obtained by excluding words irrelevant to the semantic phrase from the words in the input text , is the attention map corresponding to the given irrelevant word pair;

[0088] The negative loss function is defined between the words within the adjective and entity noun set and the words outside the set, such as the word pair'soft' 'counter', 'hard' 'towel', which is the set of mismatched words, and the negative loss function encourages to separate the word pairs with no semantic relevance, as shown in the following formula: Figure 2

[0089] The semantic loss function is obtained by adding the positive loss function and the negative loss function, as shown in the following formula:

[0090]

[0091] Step 4: Construct object-based foreground loss function and background loss function, and build region loss function based on foreground loss function and background loss function;

[0092] In order to ensure that the synthetic object in the generated image is close to the target position, that is, to ensure that the high response of cross attention is only in the mask area, based on the cross attention map and the mask set, the foreground loss function is constructed to ensure that the synthetic object is contained in the bounding box, as shown in the following formula:

[0093]

[0094] wherein, is the top elements with the highest response;

[0095] The background loss function is constructed to prevent the synthetic object in the generated image from being generated outside the target area, as shown in the following formula:​

[0096]

[0097] For all bounding boxes, the region loss function is constructed based on the foreground loss function and the background loss function, as shown in the following formula:

[0098]

[0099] wherein, is the i-th bounding box;

[0100] Step 5: Based on the semantic total loss function and the region loss function, the total loss function is established, and the latent image is iteratively updated based on the total loss function to remove the noise in the latent image, and a generated image containing a synthesized object is obtained;

[0101] The semantic loss, the region loss and the original loss of the diffusion model are combined to obtain the total loss function, as shown in the following formula:

[0102]

[0103] wherein, is a weight hyperparameter;

[0104] The latent image is updated based on the total loss function , and an updated latent image is obtained, as shown in the following formula:

[0105]

[0106] wherein, is the learning rate of the time step t, which is usually linearly decayed with the time step, and the update amplitude decreases as the denoising process proceeds, is the gradient operator of , indicating the partial derivative of the latent image ;

[0107] The noise in the latent image is removed by iterative updating to obtain a denoised latent image .

[0108] The denoised latent image is passed to the decoder to generate a generated image corresponding to the input text .

[0109] Figure 3 ​For the results of text-based image generation based on different methods, the input text is "The soft towel was on top of the hard counter", wherein (a) is the result of text-based image generation based on the Stable Diffusion model, and the semantic information of "on top of" in the sentence is not realized in the figure; (b) is the result of text-based image generation based on the Structure Diffusion, and the semantic information of "soft towel" in the sentence is lacking in the figure; (c) is the result of text-based image generation based on the Attend-and-Excite, and the semantic understanding of "soft towel" and "on top of" in the sentence is not accurate enough; (d) is the result of text-based image generation based on the present application, and the semantic information is relatively well understood, and a reasonable picture is generated.

[0110] Embodiment 2

[0111] The present embodiment provides an electronic device, comprising one or more processors, and a memory for storing instructions, when the instructions are executed by the one or more processors, the one or more processors perform the training-free text-image generation method based on the diffusion model.

[0112] The electronic device can be a mobile phone, a computer, or a tablet computer, etc., comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to realize the training-free text-image generation method based on the diffusion model as described in the embodiments. It can be understood that the electronic device can also include an input / output (I / O) interface and a communication component.

[0113] The processor is used to execute all or part of the steps in the training-free text-image generation method based on the diffusion model as described in the above embodiments. The memory is used to store various types of data, which may, for example, include instructions of any application program or method in the electronic device, and application-related data.

[0114] The processor can be an Application Specific Integrated Cricuit (ASIC), a Digital Signal Processor (DSP), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements, which are used to execute the diffusion model-based training-free text-to-image generation method described in the above embodiments.

[0115] Embodiment 3

[0116] The embodiment provides a computer readable storage medium storing executable instructions, which, when executed, can be stored in one computer readable storage medium if implemented in the form of a software function unit and sold or used as an independent product.

[0117] The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the diffusion model-based training-free text-to-image generation method described in various embodiments of the present application.

[0118] The aforementioned storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD (Secure Digital Memory Card) or a DX (Memory Data Register, MDR) memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an APP (Application) application store, and various media capable of storing a program verification code, on which a computer program is stored, which can implement each step of the diffusion model-based training-free text-to-image generation method described above when executed by a processor.

[0119] Embodiment 4:

[0120] The embodiment provides a computer program product comprising computer programs or instructions, which, when executed by a processor, implement the training-free text-to-image generation method based on a diffusion model.

[0121] Based on such understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a computer program product.

[0122] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

[0123] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various modifications and changes to the present disclosure without departing from the scope and spirit of the present disclosure. If these modifications and changes belong to the scope of the present disclosure and its equivalent technology, the present disclosure also includes these modifications and changes.

Claims

1. A training-free text-image generation method based on a diffusion model, characterized in that, Includes the following steps: Obtain the input text and use a pre-trained natural language model to extract semantic phrases and layout information from the input text; The potential image is generated in the pre-trained diffusion model StableDiffusion by taking the input text, semantic phrases and layout information as additional conditional inputs, and extracting the cross attention map of the input text at different time steps. The total loss function of the Stable Diffusion diffusion model is constructed based on the cross-attention map. The latent image is iteratively updated based on the total loss function, and noise in the latent image is removed to obtain the generated image containing the synthetic object.

2. The training-free text-image generation method based on a diffusion model according to claim 1, characterized in that, The semantic phrases and layout information in the input text are extracted using a pre-trained natural language model. The specific method is as follows: The number of words obtained is Input text , input text The input is fed into a pre-trained natural language model, which then leverages the model's understanding capabilities to interpret the input text. Perform fine-grained segmentation to extract the input text. The semantic phrases and layout information contained therein are used to obtain a set of semantic phrases and a set of layout information. The semantic phrase includes an entity noun and its modifiers, and the layout information includes the coordinates of each entity noun. Extracting input text using natural language models In The extracted semantic phrase set is represented as { }={ },in,{ } indicates that each semantic phrase is in the input text Location information, Indicates the first Entity nouns in a semantic phrase, Indicates the first Modifiers of entity nouns in a semantic phrase, express and There is a dependency relationship; Set the generated image size and use a natural language model to obtain entity names. The layout information, which consists of the position coordinates of entity names in the generated image, is obtained by performing binary mask conversion on the layout information to obtain a mask set. .

3. The training-free text-image generation method based on a diffusion model according to claim 1, characterized in that, The Stable Diffusion model includes: Variational Autoencoder (VAE), CLIP Text Encoder, and U-Net Noise Predictor; The latent image is generated in the pre-trained diffusion model StableDiffusion by using the input text, semantic phrases, and layout information as additional conditional inputs. Cross-attention maps of the input text at different time steps are extracted. The specific method is as follows: Using a variational autoencoder (VAE) to obtain data from a standard normal distribution The image is sampled from a noisy image and compressed into the latent space to obtain the latent image. ; Use CLIP text encoder to process the input text. Preprocessing is performed to obtain text tags. ; The latent image and text labels are input into the U-Net noise predictor, and cross-attention maps at different time steps are extracted. The specific method is as follows: For potential images Using a noise denoiser Iterative denoising is performed step by step to obtain intermediate features. , For potential spatial noise variables at time step t; At different time steps, based on having Text tags of 1 word and intermediate features The learning matrix for each word , Projection yields the query matrix for each word. Bond matrix Calculate the cross-attention map for each word to obtain Cross-attention diagram of words .

4. The training-free text-image generation method based on a diffusion model according to claim 1, characterized in that, The total loss function of the Stable Diffusion model, constructed based on cross-attention maps, includes: Construct object-based positive and negative loss functions to obtain the semantic loss function; Construct object-based foreground and background loss functions, and then construct a region loss function based on the foreground and background loss functions; The total loss function is established based on the semantic total loss function and the regional loss function.

5. The training-free text-image generation method based on a diffusion model according to claim 4, characterized in that, Constructing object-based positive and negative loss functions to obtain the semantic loss function, the specific method is as follows: A positive loss function is established by minimizing the distance between all entity nouns and their corresponding modifiers, as shown in the following formula: in, For the first Cross-attention graph of words, For the first Cross-attention graph of words, (This is a cross-attention map) and The similarity between them is shown in the following formula: in, KL divergence is used to calculate the distance metric between cross-attention maps; modifiers and entity nouns With input text The remaining words are compared to establish a negative loss function, as shown in the following formula: in, To obtain the input text of The set of mismatched words obtained by excluding words unrelated to the semantic phrase from a given set of words. It is an attention map corresponding to a given set of unrelated words; positive loss function and negative loss function Adding them together yields the semantic loss function. .

6. The training-free text-image generation method based on a diffusion model according to claim 4, characterized in that, Construct object-based foreground and background loss functions, and then construct a region loss function based on the foreground and background loss functions. The specific method is as follows: Based on the cross-attention map and mask set, a foreground loss function is constructed to ensure that the synthetic object is contained in the bounding box, as shown in the following formula: in, To select the highest response One element; A background loss function is constructed to prevent synthetic objects in the generated image from being generated outside the target region, as shown in the following formula: For all bounding boxes, a region loss function is constructed based on the foreground and background loss functions, as shown in the following formula: in, For the first A bounding box.

7. The training-free text-image generation method based on a diffusion model according to claim 4, characterized in that, The total loss function is established based on the semantic total loss function and the region loss function. The specific method is as follows: By combining semantic loss, region loss, and the original loss of the diffusion model, we obtain the total loss function, as shown in the following formula: in, For weight hyperparameters.

8. The training-free text-image generation method based on a diffusion model according to claim 1, characterized in that, The latent image is iteratively updated based on the total loss function, and noise in the latent image is removed to obtain the generated image containing the synthesized object. The specific method is as follows: Update the latent image based on the total loss function. To obtain the updated potential image As shown in the formula below: in, The learning rate at time step t typically decreases linearly with time step, but the update magnitude decreases as the denoising process progresses. for The gradient operator represents the gradient of the latent image. Find the partial derivatives; Noise in the latent image is removed by iterative updates, resulting in a denoised latent image. ; Denoising the latent image Passed to the decoder to generate the input text Corresponding generated image .

Images