An image generation method and apparatus based on a diffusion model

Abstract

The application discloses an image generation method and device based on a diffusion model, constructs a conditional vector generation module containing detail preservation, an attention-based diffusion model and a secondary distillation iteration framework, accelerates the running speed of the diffusion model, minimizes the influence on the quality of generated images, achieves a better balance between reducing model running loss and improving image quality, introduces a conditional attention mechanism into the diffusion model, encodes the conditional vector by using the multimodal adaptability of the self-attention mechanism, and simultaneously enhances the capture of image details, and can be used as a general algorithm framework in the field of image generation.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to an image generation method and apparatus based on a diffusion model. Background Technology

[0002] Image generation has important applications in computer vision and various real-world scenarios, such as image-text interaction, data augmentation, and artistic creation, and has become a highly popular research area in recent years. Image generation methods based on diffusion models, originating from the diffusion process in thermodynamics, have shown significant advantages in image generation, surpassing previous GAN or VAE-based methods in terms of controllability, innovation, and realism. However, the low computational efficiency of diffusion models is an unavoidable problem, especially for classifier-free diffusion schemes. This inefficiency stems not only from the step-by-step basic iteration process but also from the need to calculate noise estimations for both guided and unguided scenarios. While classifier-free diffusion models can significantly improve the quality of generated samples, this comes at the cost of high computational costs. Current solutions incorporate knowledge distillation into diffusion models to reduce the necessary number of iterations through student models, but the results are less than ideal, with a noticeable loss of image quality and the inability to generate detailed features. Therefore, achieving a better balance between improving generation efficiency and quality using diffusion models is a problem that needs to be addressed. Summary of the Invention

[0003] The purpose of this invention is to provide an image generation method and apparatus based on a diffusion model to solve the above-mentioned problems. This invention guides the diffusion model to iterate rapidly by inputting text descriptions, and finally obtains high-quality generated images.

[0004] The present invention achieves the above objectives through the following technical solutions:

[0005] An image generation method based on a diffusion model includes the following steps:

[0006] Step 1: Construct and train the teacher model by combining text encoding:

[0007] Construct a self-attention encoding module to obtain the conditional vector;

[0008] By combining the self-attention encoding module, a Transformer-based teacher diffusion model is constructed;

[0009] The teacher model was trained using implicit conditional guidance.

[0010] Step 2, perform secondary distillation on the teacher model:

[0011] Based on the teacher model, construct a student model with fewer parameters;

[0012] In a single distillation training, the trained student model is used as the new teacher model;

[0013] Construct a new student model and train it using double distillation.

[0014] Step 3: Use the final model to generate the image.

[0015] Input text conditions to generate an image.

[0016] A further approach involves the following step in step 1: constructing a self-attention encoding module to obtain the condition vector. This includes inputting a text description, i.e., information about the target generated image, and processing it to output a one-dimensional condition vector, which serves as a guiding label during the iteration process. This condition generation module is built based on an attention mechanism and includes at least one encoder and one self-attention module. The self-attention-based condition generation module uses text encoding and attention computation units to learn the semantic information in the text conditions and integrates it into the output vector.

[0017] The Transformer-based teacher diffusion model is constructed as follows: The teacher model is used to fuse conditional variables to achieve noise estimation and image generation in an iterative manner. Its backbone is divided into four layers, each with at least two attention units. The attention units integrate multi-head self-attention and variable window mechanisms to enhance information interaction between local components and adapt to feature learning and probability estimation of large-size inputs.

[0018] To ensure the semantic accuracy of the generated image, the conditional variables from the first step are input into two different locations in the model. (a) As token units, they are appended to the output of the downsampling layer, i.e., after the image token; (b) In the multi-head self-attention modules of the first two layers, the conditional vector is used as auxiliary information and concatenated with the weight sequence in the self-attention module.

[0019] Training the teacher model using implicit conditional guidance involves: inputting a noisy image; the teacher model T is used to estimate the noise to be removed, and this noise is subtracted from the input image iteratively. The training steps include three stages: parameter initialization, pre-training, and fine-tuning with target domain data. The model training methods include at least those commonly used in diffusion models, such as explicit and implicit conditional guidance.

[0020] During explicit conditional training, the noise distribution predicted by the model is as follows:

[0021]

[0022] During implicitly conditional training, the noise distribution predicted by the model is as follows:

[0023]

[0024] Where 'c' represents a conditional tag. This indicates no condition, and s is the scale coefficient used to adjust the influence weight of the condition.

[0025] The model's loss function is:

[0026] Loss=‖∈-∈ θ (x t ,t)‖ 2

[0027] Where ∈ represents the actual sampling noise, ∈ θ (x t ,t) represents the model estimation noise, and t represents the number of iterations.

[0028] A further approach is that, in step 2, constructing the student model includes: the student model inherits the learning capabilities of the teacher model, achieving output results similar to the teacher model with less computational cost. A model S1 with the same structure as the teacher model is constructed, but its number of parameters can be reduced according to task requirements and application scenarios; specifically, the number of blocks in the third level of the model is reduced.

[0029] A single distillation training step includes: initializing the student model using the parameters of the teacher model, and transferring the learning experience of the teacher model to the student model. The iteration step size of the student model should be equal to half that of the teacher model; that is, the loss function calculation of the student model after one iteration should be derived from the estimation results of the teacher model after two iterations.

[0030] The teacher model iterates through two time points, t1 and t2, and the resulting output is calculated as follows: At this point, the student model undergoes one iteration, and its output is calculated as follows: Therefore, the loss function of the student model needs to include the mean squared errors of both:

[0031]

[0032] The second distillation training involves repeating the first distillation process, shortening the iteration step size again, while ensuring that the noise estimation of the teacher model is not affected. The trained student model S1 is used as the new teacher model to construct another student model S2, whose parameters are inherited from S1. The same training method as the first distillation is adopted, but the number of iterations is reduced by half.

[0033] Using the trained second student model S2 as the final model, its parameter size can be significantly reduced, and the number of iterations can be reduced to one-quarter compared to the teacher model.

[0034] The training method of double distillation reduces the number of iterations of the diffusion model by a factor of two, and the auxiliary support based on conditional vectors preserves the details of the generated image.

[0035] This step involves at least two teacher models. During the teacher-student knowledge distillation process, the feature details learned by the teacher model are preserved, reducing the computational resources and time consumed by multiple iterations.

[0036] A further approach is that, in step 3, the trained model is used to generate the image; the input text guides the model to gradually recover the image by iterating over noise; when there is textual conditional assistance, the text information is added to the iteration process through the attention module; when there is no textual conditional assistance, the conditional variable is set as a random variable.

[0037] In another aspect, the present invention provides an image generation apparatus based on a diffusion model, comprising:

[0038] The model building module, combined with the self-attention mechanism, constructs a text-conditional information extraction module and a noise estimation module, which are applicable to feature extraction and detail preservation of generated images for both text and image modalities. It includes: a text encoding module, a self-attention extraction module, a Transformer noise distribution estimation and diffusion iteration module;

[0039] The secondary distillation module constructs at least two teacher models and uses a teacher-student distillation pattern to halve the number of iterations, thus accelerating the operation of the diffusion model. It includes: a module for constructing a model framework based on multi-head self-attention, a module for acquiring and fusing conditional variables of multiple modalities, and modules for primary and secondary distillation.

[0040] The image generation module uses the student model obtained from the last distillation to perform generation guidance, noise estimation, and iterative image generation based on text conditions;

[0041] In the image generation module, the generation process given the conditional vector is as follows:

[0042] Input the text conditions into the information extraction module in the form of sentences or words to obtain the encoded condition vector;

[0043] Using the trained second student model, diffusion iteration is performed in conjunction with the conditional vector;

[0044] The noise during the iteration process is removed from the real noise, and after a certain number of calculations, the generated image is obtained.

[0045] The text encoding module is used to process the input text information into feature vectors to assist the iterative process of the diffusion model;

[0046] The self-attention extraction module constructs a text encoder based on self-attention units to process contextual information;

[0047] The model training of the diffusion iteration module adopts the diffusion model approach, and a noise estimate is obtained in each iteration;

[0048] The backbone of the model framework module based on multi-head self-attention adopts a Transformer-based model, and noise estimation is performed based on the attention mechanism.

[0049] The conditional variable module for acquiring and fusing multimodal data adds conditional variables to specific locations in the model to guide image generation;

[0050] The primary and secondary distillation modules perform at least two consecutive distillation operations, which reduces the number of iterations by a factor of two.

[0051] The beneficial effects of this invention are as follows:

[0052] This invention uses a self-attention module as the network infrastructure, constructing a conditional vector generation module that preserves details, an attention-based diffusion model, and a double distillation iterative framework. This accelerates the operation of the diffusion model while minimizing its impact on the quality of the generated image, achieving a better balance between reducing model runtime overhead and improving image quality. By introducing the conditional attention mechanism into the diffusion model and utilizing the multimodal adaptability of the self-attention mechanism, the conditional vector is encoded, and the capture of image details is enhanced. This invention can serve as a general algorithmic framework in the field of image generation.

[0053] This invention introduces a secondary distillation method into the diffusion process. Teacher-student distillation reduces the iteration process by at least four times, significantly improving generation efficiency. Existing image generation schemes based on denoising diffusion probability models mostly employ a classifier-free approach. While this effectively improves the quality of generated images, it exacerbates the problem of low sampling efficiency. This invention combines knowledge distillation and conditional attention mechanisms with the diffusion model, using a student model to alleviate the slow iteration speed and textual conditional assistance to address the issue of image detail preservation, thereby accelerating model computation and optimizing generation quality. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1This is a flowchart of the "Image Generation Method Based on Diffusion Model" of the present invention.

[0056] Figure 2 This is a network structure diagram of the "Image Generation Method Based on Diffusion Model" of the present invention.

[0057] Figure 3 This is a schematic diagram of the "Image Generation Device Based on Diffusion Model" of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] Example 1:

[0060] Image generation is performed using the Swing Transformer model as the basic network structure. First, the conditional generation module is combined with the basic structure to build and train the teacher model. Then, two student models are built consecutively using knowledge distillation to complete two classroom-student distillation training processes. Finally, image generation is performed on the second trained student model. Figure 1 The flowchart of the present invention, "An Image Generation Method Based on a Diffusion Model," is provided, including the following steps:

[0061] 1) Model building and training, the specific steps are as follows:

[0062] Step 1: Obtain the conditional vector. A conditional generation module based on a self-attention mechanism is constructed. This module uses one RNN encoding layer and two cascaded self-attention blocks. The input to this module is text word vectors, i.e., information about the target generated image. After processing, it outputs a 1*128 length conditional vector, which serves as a guiding label during the iteration process.

[0063] Step 2: Construct the teacher model T. The backbone of the diffusion model adopts the Swin Transformer framework, and the network structure is divided into four layers, with each layer having [2, 2, 6, 2] blocks respectively. The model structure is as follows: Figure 2 As shown. The attention block integrates a multi-head self-attention mechanism and a variable window mechanism. The multi-head module sets the number of groups G. i =C i / C′, where C i Let C be the number of input channels for the i-th layer, and C′ be the number of output channels for each attention branch.

[0064] To ensure the semantic accuracy of the generated images, conditional variables are input into two different locations in the model. (a) As a token unit appended to the output of the embedding layer, i.e., after the image token; (b) In the first two layers of multi-head self-attention modules, the conditional vector is used as auxiliary information and concatenated with the weight sequence in the attention.

[0065] The third step is to train the teacher model, which includes three stages: parameter initialization, pre-training, and fine-tuning with target domain data. The model uses random parameter initialization, pre-training with a natural image dataset containing text and image pairs, and fine-tuning with a target domain dataset.

[0066] The diffusion model is trained using implicit conditional guidance. Each iteration requires calculating the noise distribution under both conditional and unconditional conditions, as shown in the following equation:

[0067]

[0068] Where 'c' represents a conditional tag. This indicates no condition, and s is the scale coefficient used to adjust the influence weight of the condition.

[0069] The model's loss function is:

[0070] Loss=‖∈-∈ θ (x t ,t)‖ 2

[0071] Where ∈ represents the actual sampling noise, ∈ θ (x t ,t) represents the model estimation noise, and t represents the number of iterations.

[0072] 2) Secondary knowledge distillation training, the specific steps are as follows:

[0073] Step 1: Construct the student model. The student model has the same structure as the teacher model, but with fewer parameters. Specifically, the number of blocks in the third level of the model is set to 4.

[0074] Step 2: Single Distillation Training. Initialize the student model using the parameters of the teacher model. The iteration step size of the student model should be equal to half that of the teacher model. That is, the loss function calculation of the student model after one iteration should be derived from the estimation results of the teacher model after two iterations.

[0075] The teacher model iterates through two time points, t1 and t2, and the resulting output is calculated as follows: At this point, the student model undergoes one iteration, and its output is calculated as follows: Therefore, the loss function of the student model needs to include the mean squared errors of both:

[0076]

[0077] Step 3: Secondary Distillation Training. Using the trained student model S1 as the new teacher model, another student model S2 is constructed. S2 inherits the parameters from S1, but the number of parameters is reduced in both the number of blocks and attention branches. The same training method as the first distillation is used, but the number of iterations is reduced by half again.

[0078] The trained second student model S2 is used as the final model.

[0079] 3) Image generation, the specific steps are as follows:

[0080] Image generation is performed using a trained model combined with textual conditions. Textual information is incorporated into the iterative process through an attention module.

[0081] Example 2:

[0082] Figure 3 A schematic diagram of the present invention, "An Image Generation Device Based on a Diffusion Model," is provided, comprising:

[0083] The model building module, combined with the self-attention mechanism, constructs a text-conditional information extraction module and a noise estimation module, which are applicable to feature extraction and detail preservation of generated images for both text and image modalities. It includes: a text encoding module, a self-attention extraction module, a Transformer noise distribution estimation and diffusion iteration module;

[0084] The text encoding module is used to process the input text information into feature vectors to assist the iterative process of the diffusion model;

[0085] The self-attention extraction module constructs a text encoder based on self-attention units to process contextual information;

[0086] The model training of the diffusion iteration module adopts the diffusion model approach, and a noise estimate is obtained in each iteration;

[0087] The secondary distillation module constructs at least two teacher models and uses a teacher-student distillation pattern to halve the number of iterations, thus accelerating the operation of the diffusion model. It includes: a module for constructing a model framework based on multi-head self-attention, a module for acquiring and fusing conditional variables of multiple modalities, and modules for primary and secondary distillation.

[0088] The backbone of the model framework module based on multi-head self-attention adopts a Transformer-based model, and noise estimation is performed based on the attention mechanism.

[0089] The conditional variable module for acquiring and fusing multimodal data adds conditional variables to specific locations in the model to guide image generation;

[0090] The primary and secondary distillation modules perform at least two consecutive distillation operations, which reduces the number of iterations by a factor of two.

[0091] The image generation module uses the student model obtained from the last distillation to perform generation guidance, noise estimation, and iterative image generation based on text conditions.

[0092] Specifically, in the image generation module, the generation process given the conditional vector is as follows:

[0093] Input the text conditions into the information extraction module in the form of sentences or words to obtain the encoded condition vector;

[0094] Using the trained second student model, diffusion iteration is performed in conjunction with the conditional vector;

[0095] The noise during the iteration process is removed from the real noise, and after a certain number of calculations, the generated image is obtained.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately. Furthermore, various different embodiments of the present invention can also be arbitrarily combined, as long as they do not violate the spirit of the present invention, they should also be considered as the content disclosed in the present invention.

Claims

1. A method for generating an image based on a diffusion model, characterized by, The method comprises the following steps: Step 1, constructing and training a teacher model in combination with text coding: constructing a self-attention coding module to obtain a condition vector; in combination with the self-attention coding module, constructing a teacher diffusion model based on a Transformer; training the teacher model in an implicit condition guiding mode; Step 2, secondary distillation of the teacher model: constructing a student model with smaller parameter quantity in reference to the teacher model; primary distillation training, taking the trained student model as a new teacher model; constructing a new student model and performing secondary distillation training; Step 3, realizing image generation by using the final model: inputting a text condition to obtain a generated image; in Step 1, constructing a self-attention coding module to obtain a condition vector comprises: inputting a text description, i.e. information about a target generated image, and outputting a one-dimensional condition vector as a guide label in an iterative process through operation; a condition generation module based on self-attention adopts a text coding and attention calculation unit to learn semantic information in a text condition and integrates the semantic information in an output vector; In step 1, the teacher model is trained in an implicit conditional guidance manner, which includes inputting a noisy image, and the teacher model The noise to be removed is estimated, and the input image is subtracted in an iterative manner; the training step includes three stages of parameter initialization, pre-training and target domain data fine-tuning, and the model training manner at least includes the training manner commonly used by the diffusion model, including explicit conditional guidance and implicit conditional guidance; when training in an explicit condition guiding mode, a noise distribution predicted by the model is as follows: ； when training in an implicit condition guiding mode, a noise distribution predicted by the model is as follows: ； wherein, is a conditional tag, denotes unconditional, is a scale coefficient for adjusting the impact weight of the condition; a loss function of the model is as follows: ； wherein, is the real sampling noise, is the model estimation noise, denotes the number of iteration steps; in Step 2, primary distillation training comprises: initializing the student model by using parameters of the teacher model to pass learning experience of the teacher model to the student model; an iterative step of the student model should be equal to half of that of the teacher model, i.e. a loss function calculation after one iteration of the student model should be derived from estimation results after two iterations of the teacher model; Teacher model goes through and two iterations of time, the output obtained is counted as At this time, the student model goes through an iteration, and its output is counted as The loss function of the student model needs to add the mean square error of the two: ； In step 2, the secondary distillation training includes: repeating the process of the first distillation, shortening the iteration step again, while ensuring that the noise estimation of the teacher model is not affected; the trained student model As a new teacher model, a student model is constructed again , The parameters of the student model are inherited from The same training method as the first distillation is adopted, and the number of iterations is reduced by half; the trained second student model as the final model.

2. The image generation method based on a diffusion model according to claim 1, wherein, in Step 1, constructing a teacher diffusion model based on a Transformer comprises: inputting the condition vector to two different positions of the model: (a) as a token unit attached to an output of a down-sampling layer, i.e. after an image token; (b) in a multi-head self-attention module of the first two levels, the condition vector is spliced with a weight sequence in the self-attention module as auxiliary information.

3. The method of claim 1, wherein the diffusion model is a diffusion tensor model. The step 2 comprises: constructing a model with the same structure as the teacher model The parameter quantity of the model can be reduced according to task requirements and application scenarios, specifically, the number of blocks in the third level of the model is reduced.

4. The image generation method based on a diffusion model according to claim 1, wherein, in Step 3, image generation is performed by using the trained model, text guidance is inputted, and the model gradually recovers an image through iterative noise; when there is text condition assistance, text information is added to an iterative process through an attention module; when there is no text condition assistance, the condition variable is set as a random variable.

5. An image generation apparatus based on a diffusion model, characterized by: The method comprises the following steps: a model construction module, in combination with a self-attention mechanism, constructs an information extraction module and a noise estimation module of a text condition, is suitable for feature extraction of two modalities of text and image and maintains details of a generated image, and comprises: a text coding module, a self-attention extraction module, a Transformer noise distribution estimation and diffusion iteration module; a secondary distillation module, at least two teacher models are constructed, a teacher-student distillation mode is used to halve an iteration number, and a diffusion model running speed is accelerated, and the secondary distillation module comprises: a model framework module based on a multi-head self-attention, a condition variable acquisition and fusion module of multiple modalities, a primary distillation and secondary distillation module; An image generation module uses the student model obtained by the last distillation to perform generation guidance, noise estimation and iterative generation of images according to the text condition; In the image generation module, the generation process under the condition of a given condition vector is as follows: The text condition is input into the information extraction module in the form of a sentence or a word, and a coded condition vector is obtained; The trained second student model is used to perform diffusion iteration combined with the condition vector; The noise in the iteration process is removed from the real noise, and after a certain number of calculations, a generated image is obtained.

6. The image generation device based on the diffusion model according to claim 5, wherein The text encoding module is used to process the input text information into a feature vector to assist the iterative process of the diffusion model; The self-attention extraction module constructs a text encoder based on a self-attention unit to process context-related information; The model training of the diffusion iteration module adopts a diffusion model, and a noise estimate is obtained at each iteration; The model backbone of the model framework module based on multi-head self-attention adopts a model based on Transformer, and noise estimation is performed according to the attention mechanism; The condition variable acquisition and fusion module adds the condition variable to a specific position of the model to guide image generation; The one-time distillation and two-time distillation module performs at least two continuous distillation operations, and the number of iterations is reduced by a factor of two.

Images