Image generation method based on multi-condition control diffusion model and electronic equipment
By extracting multiple conditions of the image and training the diffusion model using the variational autoencoder and cross-attention mechanism, the problem of poor condition management in the multi-condition generation scenario is solved, and the stability and efficiency of the generated results are improved.
Patent Information
- Application Number
- CN202411836974.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-29
AI Technical Summary
In multi-condition generation scenarios, it is difficult for existing diffusion models to effectively manage and integrate multiple input conditions, resulting in inconsistent or unreasonable generation results, and new conditions require retraining of the model, which is costly to high computing resources and time.
By extracting multiple conditions of the image, building training samples, and using variational autoencoder and cross-attention mechanisms, diffusion models are trained to manage and integrate multiple input conditions. The specific steps include: obtaining the image training set, extracting multiple conditions, training the variational autoencoder and diffusion model, and generating images using the trained model.
Effectively manage and integrate multiple input conditions to avoid inconsistent or unreasonable generation results, reduce the computing resources and time costs of model training, and improve the stability and efficiency of the model.
Smart Images
Figure CN119904541A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image generation, and in particular to an image generation method and electronic equipment based on a multi-condition controlled diffusion model. Background Art
[0002] As an advanced generative model, the diffusion probability model has shown great potential in the field of data generation in recent years. The model can generate new data that conforms to the distribution by learning the distribution characteristics from the data set. Its training process includes gradually adding noise until the data loses available information, and then reverse learning to restore the original data. This unique training method enables it to effectively capture the complex structure and statistical characteristics of the data. The diffusion model has been widely used in the fields of image generation and video generation. In order to meet complex generation needs, the diffusion model also supports conditional generation and multi-conditional generation. Conditional generation allows users to provide specific input information, such as text descriptions or labels, during the generation process to control the characteristics of the generated results. For example, users can provide a description of "a sunny beach" to guide the model to generate a related image. This technology greatly enhances the flexibility of generated content and makes the generated results more in line with user expectations.
[0003] Multi-condition generation further expands this capability, allowing the model to accept multiple input conditions at the same time to generate more complex and diverse results. For example, users can provide stick figures and text descriptions at the same time, allowing the model to generate images that conform to both the graphic features and the text content. This approach shows great potential in design, art creation, and other fields that require a high level of personalization.
[0004] Despite the remarkable results achieved in multi-conditional control generation, the diffusion model still faces some unresolved issues. For example, there may be conflicts between multiple control conditions, which may lead to inconsistent or unreasonable generation results, such as stick figures not matching the depth map outline. In addition, adding new conditions to the model usually requires retraining, which is very costly in terms of computing resources and time.
[0005] Therefore, how to effectively manage and integrate multiple input conditions and improve the stability and efficiency of the model remains the focus of current research. Summary of the invention
[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an image generation method and electronic device based on a multi-condition controlled diffusion model, so as to solve or partially solve the problem of being unable to effectively manage and integrate multiple input conditions in a multi-condition generation scenario.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] One aspect of the present invention provides an image generation method based on a multi-condition controlled diffusion model, comprising the following steps:
[0009] Step S1, obtaining an image training set, extracting multiple conditions of the image for each image data therein, and constructing a training sample;
[0010] Step S2, based on the training samples, training the variational autoencoder with the goal of minimizing the difference between the predicted conditions and the actual conditions predicted by the variational autoencoder;
[0011] Step S3, randomly selecting one or more conditions as a subset, using the variational autoencoder to calculate the encoding features corresponding to the subset, and training the diffusion model based on the encoding features and the noisy image data with the goal of minimizing the difference between the predicted noise and the actual noise;
[0012] Step S4, using the trained diffusion model to realize image generation.
[0013] As a preferred technical solution, the multiple conditions for extracting images include one or more of simple drawings, depth maps, canny pictures, low-resolution pictures, and masked pictures extracted based on the image data.
[0014] As a preferred technical solution, the stick figure and the depth map are extracted through deep learning, the canny image is obtained through edge extraction, the low-resolution image is obtained through downsampling, and the masked image is obtained by randomly generating a mask.
[0015] As a preferred technical solution, the step S2 includes:
[0016] Step S201, initializing the network weights of the variational autoencoder;
[0017] Step S202, using the training sample as the input of the variational autoencoder, obtaining the code through the encoder, multiplying the code with the mask as the input of the decoder, and obtaining the prediction condition;
[0018] Step S203, calculating the loss function value based on the predicted conditions and the actual conditions obtained in advance;
[0019] Step S204, updating the network weights of the variational autoencoder and the mask based on the loss function value;
[0020] Step S205, determine whether the variational autoencoder has been trained, if not, execute step S202, if yes, execute step S3.
[0021] As a preferred technical solution, in step S203, the loss function value is calculated using the following formula:
[0022]
[0023] Among them, L is the loss function value, MSE represents the mean square error function, represents the prediction condition, c i Indicates actual conditions.
[0024] As a preferred technical solution, the step S3 includes:
[0025] Step S301, using the trained variational autoencoder to obtain the code corresponding to each condition;
[0026] Step S302, randomly selecting one or more conditions as a subset, combining the masks of the conditions in the subset, and multiplying them with the code corresponding to the image to obtain the code feature corresponding to the subset;
[0027] Step S303, injecting control conditions into the encoding features using a cross-attention mechanism;
[0028] Step S304, inputting the noisy image data into a diffusion model to predict the noise on the image;
[0029] Step S305, calculating a loss function value based on the predicted noise on the image and the actual noise;
[0030] Step S306: Update the network parameters of the diffusion model based on the loss function value.
[0031] As a preferred technical solution, in the cross-attention mechanism, the input conditions are linearly encoded as Q values, and the image data are linearly encoded as K and V values.
[0032] As a preferred technical solution, in the step S1, after acquiring the image data set, the step further includes cropping and resizing the image data.
[0033] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned image generation method based on the multi-condition controlled diffusion model.
[0034] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned image generation method based on a multi-condition controlled diffusion model.
[0035] Compared with the prior art, the present invention has at least one of the following beneficial effects:
[0036] (1) Effectively manage and integrate multiple input conditions: When adding a new condition, the present invention only needs to train the mask corresponding to the condition, without training other parts, including the variational autoencoder and the diffusion model. The encoding of each condition is determined by the image encoding and the mask corresponding to the condition. The image encoding is shared by multiple conditions. When adding a new condition, there is no need to retrain the variational autoencoder (corresponding to the extraction of image features), but only the mask corresponding to the new condition needs to be retrained.
[0037] (2) Effective management of condition conflicts: When conditions conflict, the conflict can be resolved by properly handling the conflicting channels in the order of all conditions pre-set by the user. The encoding of each condition is determined by the image encoding and the mask corresponding to the condition. The image encoding is shared by multiple conditions, while the mask is unique to each condition. The mask has only 0 and 1 values, which defines the combination of multiple conditions, namely, mask union, to avoid conflicts among multiple conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a flow chart of an image generation method based on a multi-condition controlled diffusion model in an embodiment;
[0039] Figure 2 A schematic diagram of a cross-attention mechanism in an embodiment;
[0040] Figure 3 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0042] Example 1
[0043] Regarding the problems existing in the above prior art, see Figure 1 This embodiment provides an image generation method based on a multi-condition controlled diffusion model, comprising the following steps:
[0044] Step S1, obtaining an image training set, extracting multiple conditions of the image for each image data therein, and constructing a training sample.
[0045] Specifically, after obtaining the image training set, the image training set is traversed to extract multiple conditions corresponding to each image in the training set, where the conditions may include one or more of the following conditions:
[0046] (1) Use the existing deep learning method PiDiNet to extract the simple sketch c1 and the existing deep learning method MiDas to extract the depth map c2.
[0047] (2) Use the traditional edge extraction algorithm to extract the canny image c3.
[0048] (3) Use downsampling to obtain the low-resolution image c4.
[0049] (4) Randomly generate a mask and obtain the masked image c5.
[0050] Step S2, based on the training samples, the variational autoencoder is trained with the goal of minimizing the difference between the predicted conditions and the actual conditions predicted by the variational autoencoder.
[0051] The variational autoencoder in this step adopts the existing structure. On this basis, multiple linear layers are added as adapters at the input of the encoder and the output of the decoder, corresponding to the number of conditions. The condition is first input to the encoder's adapter, then the code z is obtained through the encoder's public part, then decoded with the corresponding decoder's public part, and finally reconstructed through its corresponding decoder Adapter. Each condition c i Corresponding to a learnable mask m i , the mask value can only be 0 and 1. Each condition c fed to the decoder i The corresponding features are given by m i ×z is obtained. Set the termination condition of training, such as the total number of training steps.
[0052] Specifically, step S2 may include steps S201-S206.
[0053] Step S201, initializing the network weights of the variational autoencoder.
[0054] Step S202, image data is obtained from the image training set for preprocessing, and the image is processed into a preset size by resizing and cropping, and different conditions are extracted and combined to form training samples.
[0055] Step S203: input different conditions of the training samples into the variational autoencoder and reconstruct the prediction c i .
[0056] Step S204, calculate the loss function:
[0057]
[0058] In the formula, represents the prediction condition, c i represents the training condition, MSE represents the mean square error function;
[0059] Step S205, using the training error, through back propagation and an appropriate optimization algorithm (such as stochastic gradient descent), the network weights of the variational autoencoder and the learnable mask m corresponding to each condition are updated. i .
[0060] Step S206, determine whether the variational autoencoder training is completed, if so, save the neural network model and execute step S3, if not, return to S202.
[0061] Step S3, randomly select one or more conditions as a subset, use the variational autoencoder to calculate the encoding features corresponding to the subset, and train the diffusion model based on the encoding features and the noisy image data with the goal of minimizing the difference between the predicted noise and the actual noise.
[0062] In this step, the existing diffusion model Unet architecture is used, and on this basis, the cross-attention mechanism is used to inject control conditions. Initialize the weights and set the conditions for training termination, such as the total number of training steps.
[0063] Specifically, step S3 may include steps S301-306.
[0064] Step S301, obtain image data from the image training set for preprocessing, and process the image to a preset size by resizing and cropping, and then extract different conditions and combine them to form training samples.
[0065] Step S302: Input the images in the training set into the variational autoencoder trained in step S2, encode the images and generate masks m according to different conditions. i Obtain the codes of different conditions. This step is the core step of this method to achieve effective management and integration of multiple input conditions and condition conflict management.
[0066] Step S303, randomly select one or more conditions from all the conditions in step S1 as a subset, merge the masks of the subset conditions, and then multiply them with the image code to obtain the features corresponding to the subset, and use the step cross attention mechanism to send them to the generation model.
[0067] For details, see Figure 2First, the condition is encoded with a learnable linear layer to obtain Q. The feature of the image in UNet is X1, and it is encoded with a linear layer to obtain K and V. Then the matrix product of Q and K is calculated, and a softmax operation is performed. Finally, the result is matrix-multiplied with V. The feature X2 is obtained, and the subsequent steps are continued.
[0068] Step S304, add noise to the image, input it into the network, and predict the noise on the loaded image.
[0069] Step S305, calculating the MSE loss between the predicted noise and the actual noise.
[0070] Step S306: Update network parameters using the loss function.
[0071] Step S4, using the trained diffusion model to realize image generation.
[0072] This method has the following advantages:
[0073] (1) When a new condition is added, only the mask corresponding to the condition needs to be trained, without training other parts, including the variational autoencoder and the generative model.
[0074] (2) When conditions conflict, the conflict can be resolved by properly handling the conflicting channels according to the priority set by the user in advance (the user can set the order of all conditions described in step S1 according to his / her preference, and the conditions with higher priority selected by the user will be given priority during generation).
[0075] Example 2
[0076] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the image generation method based on the multi-condition controlled diffusion model as described in Example 1.
[0077] like Figure 3 As shown, at the hardware level, the electronic device may include a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The image generation method based on the multi-condition controlled diffusion model. Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0078] Example 3
[0079] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the image generation method based on the multi-condition controlled diffusion model as described in Example 1.
[0080] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0081] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. An image generation method based on a multi-condition controlled diffusion model, characterized in that: The steps include: Step S1, obtaining an image training set, extracting multiple conditions of the image for each image data therein, and constructing a training sample; Step S2, based on the training samples, training the variational autoencoder with the goal of minimizing the difference between the predicted conditions and the actual conditions predicted by the variational autoencoder; Step S3, randomly selecting one or more conditions as a subset, using the variational autoencoder to calculate the encoding features corresponding to the subset, and training the diffusion model based on the encoding features and the noisy image data with the goal of minimizing the difference between the predicted noise and the actual noise; Step S4, using the trained diffusion model to realize image generation.
2. The image generation method based on a multi-condition controlled diffusion model according to claim 1, characterized in that: The multiple conditions for extracting the image include one or more of a simple drawing, a depth map, a canny picture, a low-resolution picture, and a masked picture extracted based on the image data.
3. The image generation method based on a multi-condition controlled diffusion model according to claim 2, characterized in that: The stick figure and the depth map are extracted through deep learning, the canny image is obtained through edge extraction, the low-resolution image is obtained through downsampling, and the masked image is obtained by randomly generating a mask.
4. The image generation method based on a multi-condition controlled diffusion model according to claim 1, characterized in that: The step S2 comprises: Step S201, initializing the network weights of the variational autoencoder; Step S202, using the training sample as the input of the variational autoencoder, obtaining the code through the encoder, multiplying the code with the mask as the input of the decoder, and obtaining the prediction condition; Step S203, calculating the loss function value based on the predicted conditions and the actual conditions obtained in advance; Step S204, updating the network weights of the variational autoencoder and the mask based on the loss function value; Step S205, determine whether the variational autoencoder has been trained, if not, execute step S202, if yes, execute step S3.
5. The image generation method based on the multi-condition controlled diffusion model according to claim 4 is characterized in that: In step S203, the loss function value is calculated using the following formula: Among them, L is the loss function value, MSE represents the mean square error function, represents the prediction condition, c i Indicates actual conditions.
6. The image generation method based on a multi-condition controlled diffusion model according to claim 1, characterized in that: The step S3 comprises: Step S301, using the trained variational autoencoder to obtain the code corresponding to each condition; Step S302, randomly selecting one or more conditions as a subset, combining the masks of the conditions in the subset, and multiplying them with the code corresponding to the image to obtain the code feature corresponding to the subset; Step S303, injecting control conditions into the encoding features using a cross-attention mechanism; Step S304, inputting the noisy image data into a diffusion model to predict the noise on the image; Step S305, calculating a loss function value based on the predicted noise on the image and the actual noise; Step S306: Update the network parameters of the diffusion model based on the loss function value.
7. The image generation method based on a multi-condition controlled diffusion model according to claim 6, characterized in that: In the cross-attention mechanism, the input conditions are linearly encoded as Q values, and the image data are linearly encoded as K and V values.
8. The image generation method based on a multi-condition controlled diffusion model according to claim 1, characterized in that: In the step S1, after the image data set is acquired, the image data is cropped and resized.
9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the image generation method based on the multi-condition controlled diffusion model as claimed in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that: The method comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing the image generation method based on a multi-condition controlled diffusion model as claimed in any one of claims 1 to 8.