Image generation method based on diffusion model

Abstract

The embodiment of the invention discloses an image generation method based on a diffusion model, and the method comprises the steps: obtaining a first clean image, inputting the first clean image into a diffusion process of the diffusion model, obtaining a noise image, constructing a loss function of the diffusion model based on the features of the five sense organs of a human face, inputting the noise image into a reconstruction process of the diffusion model, and obtaining the noise image. And training the diffusion model to obtain a target diffusion model, obtaining a random Gaussian noise image, and inputting the random Gaussian noise image into a reconstruction process of the target diffusion model to obtain a generated face image. A Wavelet block is introduced into a diffusion model, the Wavelet block can be used for extracting a first frequency feature of a first feature map, namely extracting high-frequency and low-frequency information of a noise image, and the diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, so that the Gaussian noise can be effectively separated, and the quality of the generated face image is improved; and the loss function is constructed based on the features of the five sense organs of the human face, so that the quality of detail parts of the five sense organs of the generated human face image is improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing and computer vision, and in particular to an image generation method based on a diffusion model. Background Art

[0002] In recent years, face image generation technology has developed rapidly. At present, typical models for face image generation include GAN, VAE, and diffusion model. GAN has a wide range of applications and can generate high-fidelity and diverse face images with great flexibility, but the GAN training process is unstable and prone to mode collapse. The VAE training process is stable and not prone to mode collapse, but the face images generated by VAE have low fidelity and blurred details. In most cases, the generated face images have a strong sense of synthesis. The diffusion model training process is stable, and the generated face images are high-fidelity, diverse, and flexible. However, the face images generated by the diffusion model are of low quality and blurred details. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The disclosed embodiment provides an image generation method based on a diffusion model, which can improve the overall quality of the generated face image and make the generated face image more realistic and clear.

[0005] An embodiment of the first aspect of the present disclosure is a method for generating an image based on a diffusion model, comprising:

[0006] Get the first clean image in the face dataset;

[0007] The first clean image is input into the diffusion process of the diffusion model, and Gaussian noise is gradually added in the diffusion process to obtain a noisy image; the diffusion model includes a time step embedding vector generation module, an encoder, and a decoder, the decoder includes a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module includes a Wavelet block, an Attention block, the Wavelet block, and an upsampling layer connected in sequence, the decoder second module includes a ResBlock block, the Attention block, and the upsampling layer connected in sequence, the decoder third module includes the ResBlock block, the Attention block, and the upsampling layer connected in sequence, the decoder fourth module includes the ResBlock block, the upsampling layer, and the ResBlock block connected in sequence, and the Wavelet block is used to extract the first frequency feature of the first feature map through a discrete wavelet transform module;

[0008] Constructing the loss function of the diffusion model based on facial features;

[0009] Inputting the noise image into a reconstruction process of the diffusion model, training the diffusion model in the reconstruction process based on the loss function to obtain a target diffusion model;

[0010] A random Gaussian noise image is acquired, and the random Gaussian noise image is input into the reconstruction process of the target diffusion model to obtain a generated face image.

[0011] According to some embodiments of the first aspect of the present disclosure, the step of inputting the first clean image into a diffusion process of a diffusion model, and gradually adding Gaussian noise in the diffusion process to obtain a noisy image includes:

[0012] The first clean image is input into the diffusion process of the diffusion model, and the Gaussian noise is gradually added through a diffusion formula in the diffusion process to obtain the noisy image. The diffusion formula is:

[0013]

[0014] In the formula, α t ∈(0,1), x0 is the first clean image, x t is the noise image after adding the Gaussian noise for t steps, ε is the Gaussian noise, and t∈(1,T).

[0015] According to some embodiments of the first aspect of the present disclosure, the inputting the noise image into the reconstruction process of the diffusion model, and training the diffusion model in the reconstruction process based on the loss function, comprises:

[0016] The noisy image is input into the reconstruction process of the diffusion model. Based on the loss function, the Gaussian noise added to the noisy image is gradually inferred through a reconstruction formula during the reconstruction process, and the added Gaussian noise is gradually removed to obtain a second clean image. The reconstruction formula is:

[0017]

[0018] In the formula, x t is the noise image after adding the Gaussian noise after t steps, x t-1 is the noise image after adding the Gaussian noise in step t-1, α t ∈(0,1), t∈(1,T),ε θ (x t ,t) is the Gaussian noise added under the inferred parameter θ.

[0019] According to certain embodiments of the first aspect of the present disclosure, the loss function of constructing the diffusion model based on facial features includes:

[0020] Constructing an additional MSE loss based on the facial features;

[0021] The loss function of the diffusion model is composed of the mean square error loss MSE and the additional MSE loss. The loss function of the diffusion model is:

[0022]

[0023] Where, L is the loss function of the diffusion model, L MSE is the mean square error loss MSE, L attribute is the additional MSE loss.

[0024] According to certain embodiments of the first aspect of the present disclosure, constructing the additional MSE loss based on the facial features includes:

[0025] Performing feature extraction on the first clean image to obtain facial features and non-facial features of the human face;

[0026] Performing an expansion operation on the facial features to cover all areas within a preset range around the facial features, and performing the expansion operation on the non-facial features to cover edge areas;

[0027] Combining the facial features and the non-facial features after the dilation operation into a mask image;

[0028] Performing a blur operation on the mask image to obtain a target mask image;

[0029] According to the target mask image, the additional MSE loss is obtained:

[0030]

[0031] Where x0 is the first clean image, x t is the noise image after adding the Gaussian noise after t steps, ε is the Gaussian noise, ε θ (x t ,t) is the added Gaussian noise inferred under the θ parameter, M is the target mask image, t∈(1,T).

[0032] According to some embodiments of the first aspect of the present disclosure, the mean square error loss MSE is:

[0033]

[0034] Where x0 is the first clean image, x t is the noise image after adding the Gaussian noise after t steps, ε is the Gaussian noise, ε θ (x t ,t) is the inferred Gaussian noise added under the θ parameter.

[0035] According to certain embodiments of the first aspect of the present disclosure, the Wavelet block includes the discrete wavelet transform module, the ResBlock block, and the inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map, the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain a second frequency feature, and the inverse discrete wavelet transform module is used to convert the second frequency feature into a second feature map.

[0036] According to certain embodiments of the first aspect of the present disclosure, the time step embedding vector generation module includes a linear layer, a SiLU activation function, and the linear layer connected in sequence; the encoder includes a first encoder module, a second encoder module, a third encoder module, and a fourth encoder module connected in sequence, the first encoder module includes the ResBlock block and the downsampling layer connected in sequence, the second encoder module includes the ResBlock block and the downsampling layer connected in sequence, the third encoder module includes the ResBlock block, the Attention block, and the downsampling layer connected in sequence, and the fourth encoder module includes the ResBlock block, the Attention block, and the downsampling layer connected in sequence.

[0037] An embodiment of the second aspect of the present disclosure is an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, an image generation method based on a diffusion model as described above is implemented.

[0038] An embodiment of the third aspect of the present disclosure is a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method based on the diffusion model as described above is implemented.

[0039] The disclosed embodiments include at least the following beneficial effects: obtaining a first clean image in a face data set; inputting the first clean image into a diffusion process of a diffusion model, and gradually adding Gaussian noise in the diffusion process to obtain a noisy image; the diffusion model includes a time step embedding vector generation module, an encoder, and a decoder; the decoder includes a first decoder module, a second decoder module, a third decoder module, and a fourth decoder module connected in sequence; the first decoder module includes a Wavelet block, an Attention block, a Wavelet block, and an upsampling layer connected in sequence; the second decoder module includes a ResBlock block, an Attention block, an upsampling layer connected in sequence The third module of the decoder includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The fourth module of the decoder includes a ResBlock block, an upsampling layer, and a ResBlock block connected in sequence. The Wavelet block is used to extract the first frequency feature of the first feature map through a discrete wavelet transform module; the loss function of the diffusion model is constructed based on the facial features, the noise image is input into the reconstruction process of the diffusion model, and the diffusion model is trained in the reconstruction process based on the loss function to obtain a target diffusion model; a random Gaussian noise image is obtained, and the random Gaussian noise image is input into the reconstruction process of the target diffusion model to obtain a generated face image. The Wavelet block is introduced into the diffusion model, and the Wavelet block can be used to extract the first frequency feature of the first feature map, that is, to extract the high-frequency and low-frequency information of the noise image. The diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, and can effectively separate Gaussian noise. The Wavelet block has a high accuracy in extracting the frequency information of the noise image, which can improve the accuracy of separating Gaussian noise, improve the overall quality of the generated face image, and make the generated face image more realistic and clear. In addition, a loss function of the diffusion model is constructed based on the facial features. The diffusion model is trained during the reconstruction process based on the loss function to guide the diffusion model to pay more attention to the details of the facial features and improve the quality of the facial details of the generated facial image.

[0040] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0042] Figure 1 An optional flow chart of an image generation method based on a diffusion model provided in an embodiment of the present disclosure;

[0043] Figure 2 A schematic diagram of a diffusion network structure provided by an embodiment of the present disclosure;

[0044] Figure 3 A schematic diagram of the diffusion process structure provided by an embodiment of the present disclosure;

[0045] Figure 4 A schematic diagram of the reconstruction process structure provided by an embodiment of the present disclosure;

[0046] Figure 5 A schematic diagram of the Wavelet block structure provided in an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram of generating a face image from a random Gaussian noise image provided by an embodiment of the present disclosure;

[0048] Figure 7 A schematic diagram of obtaining a target mask image based on facial features provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0050] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0052] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0053] Mean Squared Error (MSE) is a commonly used loss function, which is often used to evaluate the difference between the model's predicted value and the true value. It measures the model's prediction performance by calculating the average of the squared errors between the predicted value and the true value. The smaller the mean squared error, the more accurate the model's prediction; the larger the mean squared error, the greater the model's prediction error.

[0054] Discrete wavelet transform (DWT) discretizes the scale and translation of basic wavelets. It can decompose signals or images into components of different frequencies.

[0055] Inverse discrete wavelet transform (IDWT) is the inverse process of discrete wavelet transform, which combines the frequency bands decomposed by discrete wavelet transform into the original signal.

[0056] The embodiments of the present disclosure are further described below in conjunction with the accompanying drawings.

[0057] Reference Figure 1 As shown, Figure 1 An optional flow chart of an image generation method based on a diffusion model provided in an embodiment of the present disclosure, the image generation method based on a diffusion model includes but is not limited to steps 101 to 105, and each step is introduced in turn below.

[0058] Step 101: Obtain the first clean image in the face dataset.

[0059] Specifically, the face dataset may be a CelebAMask-HQ dataset, a Flickr-Faces-Hight-Quality dataset, or a Landmark guided face Parsing dataset. The face dataset may be selected according to actual conditions, and this embodiment does not impose any specific restrictions thereon. The first clean image refers to a clear, high-quality face image that is not contaminated or disturbed by noise.

[0060] Step 102: Input the first clean image into the diffusion process of the diffusion model, and gradually add Gaussian noise during the diffusion process to obtain a noisy image.

[0061] Specifically, the diffusion process of the diffusion model refers to gradually adding noise to the original data to transform the original data into noise data.

[0062] In some embodiments, the first clean image is input into a diffusion process of a diffusion model, and Gaussian noise is gradually added during the diffusion process to obtain a noisy image, including:

[0063] The first clean image is input into the diffusion process of the diffusion model, and Gaussian noise is gradually added through the diffusion formula during the diffusion process to obtain a noisy image. The diffusion formula is:

[0064]

[0065] In the formula, α t ∈(0,1), x0 is the first clean image, x t is the noise image after adding Gaussian noise after t steps, ε is Gaussian noise, t∈(1,T). α t is the coefficient for controlling the intensity of Gaussian noise, N is the standard Gaussian distribution, and I is the variance.

[0066] Reference Figure 3 As shown, Figure 3 Schematic diagram of the diffusion process structure provided by the embodiment of the present disclosure. The first clean image x0 is input into the diffusion process of the diffusion model, and Gaussian noise is added to the first clean image x0 during the diffusion process. The diffusion process has a total of T steps, and Gaussian noise is added at each step of the diffusion process. In each step of the diffusion process, Gaussian noise is added according to the diffusion formula. The noise image obtained after adding Gaussian noise at the tth step is x t .

[0067] Specifically, Gaussian noise refers to a type of noise whose probability density function follows a normal distribution. It is understandable that Gaussian noise is a type of noise that is widely present in nature. Image data in real scenes are mostly affected by Gaussian noise. Adding Gaussian noise to the first clean image can more accurately simulate the impact on image data in real scenes, improve the robustness of the diffusion model, and enable the diffusion model to better adapt to and handle changes and noise in the real world.

[0068] Step 103: Construct a loss function of the diffusion model based on the facial features.

[0069] Specifically, the loss function is an important concept in machine learning and deep learning, which is used to measure the difference or error between the model's prediction results and the true results. It is a numerical evaluation metric that provides a measure of the model's performance by comparing the model output with the true label. When training a diffusion model, the parameters of the diffusion model are adjusted to minimize the loss function and improve the performance of the diffusion model.

[0070] In some embodiments, constructing a loss function of a diffusion model based on facial features includes: constructing an additional MSE loss based on facial features; combining the mean square error loss MSE and the additional MSE loss to form a loss function of the diffusion model, and the loss function of the diffusion model is:

[0071]

[0072] Where L is the loss function of the diffusion model, L MSE is the mean square error loss MSE, L attribute is the additional MSE loss. λ is the weight parameter.

[0073] It can be understood that on the basis of using the mean square error loss MSE, an additional MSE loss based on the facial features is added to obtain the loss function of the diffusion model. Based on the loss function, the diffusion model is trained during the reconstruction process to guide the diffusion model to pay more attention to the details of the facial features and improve the quality of the facial details of the generated face image.

[0074] Reference Figure 7 As shown, Figure 7 A schematic diagram of obtaining a target mask image based on facial features provided in an embodiment of the present disclosure. In some embodiments, constructing an additional MSE loss based on facial features includes:

[0075] Perform feature extraction on the first clean image to obtain facial features and non-facial features;

[0076] Performing an expansion operation on facial features to cover all areas within a preset range around the facial features, and performing an expansion operation on non-facial features to cover edge areas;

[0077] Combine the facial features and non-facial features after the dilation operation into a mask image;

[0078] Perform blur operation on the mask image to obtain the target mask image;

[0079] According to the target mask image, the additional MSE loss is obtained:

[0080]

[0081] Where x0 is the first clean image, x t is the noise image after adding Gaussian noise after t steps, ε is Gaussian noise, ε θ (x t ,t) is the Gaussian noise added under the θ parameter, M is the target mask image, t∈(1,T). E is the expected value.

[0082] Specifically, facial features refer to the elements that constitute the key parts of the face, such as eyes, nose, mouth, and ears, and non-facial features refer to the features of the first clean image extracted except for the facial features. Dilation operation refers to a morphological operation that expands the boundaries of objects in an image in image processing, and blur operation refers to a technique used in image processing to reduce image noise and details to make the image look softer and smoother. Mask image refers to a facial image formed by reassembling the facial features and non-facial features after the dilation operation.

[0083] It is understandable that the first clean image is feature extracted to obtain facial features and non-facial features, and the facial features are expanded to cover all areas within a preset range around the facial features to emphasize all relevant information within the preset range around the facial features. The non-facial features are expanded to cover the edge areas of the non-facial features. By covering the edge areas, it is ensured that the edge information is taken into consideration when calculating the loss, thereby improving the diffusion model's ability to retain and reconstruct the image edge. The preset range can be set according to actual conditions, and this embodiment does not impose specific restrictions on it.

[0084] It can be understood that the facial features and non-facial features after the expansion operation are combined into a mask image, and the mask image is blurred to make the edges smoother and the weight distribution smoother, thereby reducing the possibility of mutations or discontinuities.

[0085] It is understandable that after obtaining the target mask image, the additional MSE loss is obtained according to the target mask image, and the mean square error loss MSE and the additional MSE loss are combined to form the loss function of the diffusion model. By adding the target mask image to the loss function of the diffusion model, the weight of the facial features is greater during the training of the diffusion model, guiding the diffusion model to pay more attention to the details of the facial features area, so that the diffusion model is trained in the direction of less loss of facial features, better retaining the facial details and features, and improving the quality of the facial features details of the generated face image.

[0086] In some embodiments, feature extraction is performed on the first clean image to obtain facial features and non-facial features, where the facial features include eye features, and the eye features are expanded to cover all areas within a preset range around the eye features. The preset range may be an area range that includes the shape and texture information of the eye sockets to emphasize the shape and texture information of the eye sockets around the eyes.

[0087] In some embodiments, the mean square error loss MSE is:

[0088]

[0089] In the formula, x0 is the first clean image, x t is the noise image after adding Gaussian noise after t steps, ε is Gaussian noise, ε θ (x t ,t) is the estimated Gaussian noise added under the parameter θ. E is the expected value.

[0090] Specifically, the mean square error loss MSE provides a quantitative measure for the accuracy of the diffusion model prediction by calculating the average of the squares of the differences between the predicted values ​​and the true values. This measurement method is simple and intuitive. The calculation formula of the mean square error loss MSE is simple and clear, easy to implement, and has high computational efficiency, which improves the efficiency of training the diffusion model.

[0091] Step 104: Input the noise image into the reconstruction process of the diffusion model, train the diffusion model based on the loss function during the reconstruction process, and obtain a target diffusion model.

[0092] Specifically, the reconstruction process of the diffusion model is to gradually perform denoising operations on the noise data to restore the noise data to the original data.

[0093] In some embodiments, the noisy image is input into a reconstruction process of a diffusion model, and the diffusion model is trained during the reconstruction process based on a loss function, including:

[0094] The noisy image is input into the reconstruction process of the diffusion model. Based on the loss function, the Gaussian noise added to the noisy image is gradually inferred through the reconstruction formula during the reconstruction process, and the added Gaussian noise is gradually removed to obtain the second clean image. The reconstruction formula is:

[0095]

[0096] In the formula, σ t is a constant, x t is the noise image after adding Gaussian noise after t steps, x t-1 is the noise image after adding Gaussian noise after step t-1, α t ∈(0,1), t∈(1,T),ε θ (x t ,t) is the Gaussian noise added under the θ parameter. θ is the neural network parameter to be trained, that is, the parameter to be trained for the diffusion network of the diffusion model. ε is the Gaussian noise, α t is the coefficient for controlling the intensity of Gaussian noise, N is the standard Gaussian distribution, and I is the variance.

[0097] Reference Figure 4 As shown, Figure 4 A schematic diagram of the reconstruction process structure provided by the embodiment of the present disclosure. tis the noise image after adding Gaussian noise after t steps, x t-1 is the noise image after adding Gaussian noise after t-1 steps, ε θ (x t ,t) is the Gaussian noise added in the tth step under the θ parameter, ε θ (x t-1 ,t-1) is the Gaussian noise added in the t-1th step inferred under the θ parameter.

[0098] Specifically, the first clean image is input into the diffusion process of the diffusion model, and Gaussian noise is gradually added through the diffusion formula during the diffusion process to obtain a noisy image. The noisy image is input into the reconstruction process of the diffusion model, and the Gaussian noise added to the noisy image is gradually inferred through the reconstruction formula during the reconstruction process, and the added Gaussian noise is gradually removed to obtain a second clean image. Among them, the second clean image obtained after denoising is the first clean image before denoising. It can be understood that the diffusion model can improve the quality of the face image generated by the diffusion model by gradually adding Gaussian noise to the first clean image and then learning how to reverse this process to generate high-quality data. And by adding Gaussian noise and training the diffusion model to remove Gaussian noise, the robustness of the diffusion model to the input data is enhanced, so that the diffusion model can perform more robustly when processing input data with Gaussian noise.

[0099] Step 105: Obtain a random Gaussian noise image, and input the random Gaussian noise image into the reconstruction process of the target diffusion model to obtain a generated face image.

[0100] Specifically, the random Gaussian noise image refers to a face image covered by Gaussian noise of different intensities. It can be understood that the random Gaussian noise image is obtained, the random Gaussian noise image is input into the reconstruction process of the target diffusion model, the Gaussian noise in the random Gaussian noise image is gradually inferred during the reconstruction process, and the Gaussian noise is gradually removed to obtain the generated face image.

[0101] Reference Figure 6 As shown, Figure 6 A schematic diagram of a face image generated from a random Gaussian noise image provided by an embodiment of the present disclosure, Figure 6 The reconstruction process of the random Gaussian noise image input to the target diffusion model in the middle left figure is obtained Figure 6 The generated face image in the middle right image.

[0102] Specifically, the diffusion model includes a time step embedding vector generation module, an encoder, and a decoder. The decoder includes a first decoder module, a second decoder module, a third decoder module, and a fourth decoder module connected in sequence. The first decoder module includes a Wavelet block, an Attention block, a Wavelet block, and an upsampling layer connected in sequence. The second decoder module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The third decoder module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The fourth decoder module includes a ResBlock block, an upsampling layer, and a ResBlock block connected in sequence. The Wavelet block is used to extract the first frequency feature of the first feature map through a discrete wavelet transform module.

[0103] In some embodiments, the time step embedding vector generation module includes a linear layer, a SiLU activation function, and a linear layer connected in sequence; the encoder includes a first encoder module, a second encoder module, a third encoder module, and a fourth encoder module connected in sequence, the first encoder module includes a ResBlock block and a downsampling layer connected in sequence, the second encoder module includes a ResBlock block and a downsampling layer connected in sequence, the third encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence, and the fourth encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence.

[0104] Among them, ResBlock refers to Residual Block. SiLU (Sigmoid Linear Unit) activation function is an adaptive activation function. The time step embedding vector generation module is used to generate the time step embedding vector according to the current time. The Attention block is used to weight the input feature map. The linear layer, also known as the fully connected layer, is a layer used to implement linear transformation in a neural network. The upsampling layer is a technique used in deep learning to increase the resolution or size of data. The downsampling layer, also known as the pooling layer, is used to reduce the dimension and computation of the data while retaining the main features. Specifically, the encoder is responsible for compressing the input noisy image into a low-dimensional representation and extracting key features; the decoder is responsible for reconstructing the low-dimensional representation output by the encoder back to the size of the original noisy image and separating Gaussian noise in the process; the time step embedding vector generation module is responsible for encoding the time step information into a vector form as a supplementary feature of the original sequence data, enhancing the diffusion model's understanding and prediction capabilities for time series data. The time step embedding vector generation module takes the current time t as input and outputs the time step embedding vector. The time step embedding vector and the noisy image are input to the encoder as the input data of the encoder. After the noisy image is processed by the encoder, it is input to the decoder together with the time step embedding vector as the input data of the decoder. The decoder separates the Gaussian noise and outputs the second clean image after denoising. Through the collaborative work of the time step embedding vector generation module, the encoder, and the decoder, the Gaussian noise in the noisy image can be effectively predicted and removed.

[0105] In the encoder, the ResBlock block learns the input feature map and the time step embedding vector, adjusts the extraction of noise information according to different time steps, and helps the diffusion model understand the time series data. At the same time, the jump connection structure of the ResBlock block makes the diffusion model training process more stable; the Attention block weights the input feature map, so that the diffusion model pays more attention to the facial features in the noise image, and can also highlight the noise area in the noise image, so as to remove Gaussian noise more effectively. The downsampling layer reduces the dimension of the data by reducing the size of the feature map while retaining important information.

[0106] In the decoder, the upsampling layer is used to enlarge the size of the feature map. The first module of the decoder contains four blocks. The first block is the Wavelet block, which is used to extract the first frequency feature of the first feature map through the discrete wavelet transform module. The third block is the Wavelet block, which is used to convert the processed first frequency feature back to the spatial domain. These two Wavelet blocks refine the features of the first feature map and effectively separate Gaussian noise. The second module of the decoder and the third module of the decoder further refine the features of the feature map and improve the image quality. The fourth module of the decoder fine-tunes and integrates the features of the feature map.

[0107] In some embodiments, reference Figure 5 As shown, Figure 5 A schematic diagram of the Wavelet block structure provided for an embodiment of the present disclosure. The Wavelet block includes a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map, the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain the second frequency feature, and the inverse discrete wavelet transform module is used to convert the second frequency feature into a second feature map.

[0108] Specifically, the first feature map, the second feature map, and the feature map refer to a series of intermediate representations used to generate the final output, i.e., the second clean image, during the process of processing the noisy image by the encoder or decoder. The first feature map, the second feature map, and the feature map contain key information of the noisy image, and features learned by the encoder or decoder in the process of gradually removing Gaussian noise and restoring the details of the second clean image. The first frequency feature refers to features in different frequency ranges extracted after dividing the noisy image into different frequency bands by discrete wavelet transform, and the second frequency feature refers to features after separating the Gaussian noise from the first frequency feature.

[0109] In the decoder, the Wavelet block extracts the first frequency feature of the first feature map through the discrete wavelet transform module, separates the Gaussian noise according to the frequency of the first frequency feature through the ResBlock block to obtain the second frequency feature, and converts the second frequency feature into the second feature map through the inverse discrete wavelet transform module. Unlike the ResBlock block of the encoder, the ResBlock block of the decoder also needs to receive the feature map of the same size from the ResBlock block in the encoder as input, that is, the encoder and decoder pass the input directly to the output through the jump connection of the ResBlock block, which helps solve the gradient disappearance problem and improves the training stability of the diffusion model.

[0110] It can be understood that a preset frequency range is set according to the frequency range in which the Gaussian noise added to the noise image exists to separate the Gaussian noise. After the discrete wavelet transform module extracts the first frequency feature of the first feature map, the ResBlock block separates the Gaussian noise of the first frequency feature according to the preset frequency range to obtain the second frequency feature, and the inverse discrete wavelet transform module converts the second frequency feature into the second feature map. The preset frequency range can be set according to actual conditions, and this embodiment does not impose specific restrictions on it. Specifically, a Wavelet block is introduced into the diffusion model, and the Wavelet block can be used to extract the first frequency feature of the first feature map, that is, to extract the frequency information of the noise image. The ResBlock block in the Wavelet block separates the Gaussian noise according to the frequency information and the preset frequency range, which can effectively separate the Gaussian noise, and the Wavelet block has a high accuracy in extracting the frequency information of the noise image, which can improve the accuracy of separating the Gaussian noise, improve the overall quality of the generated face image, and make the generated face image more realistic and clear.

[0111] The model refers to the combination of network structure and parameters. The diffusion model includes the diffusion network, Figure 2 As shown, Figure 2 A schematic diagram of a diffusion network structure provided for an embodiment of the present disclosure. The diffusion network includes a time step embedding vector generation module, an encoder, and a decoder. The decoder includes a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence. The decoder first module includes a Wavelet block, an Attention block, a Wavelet block, and an upsampling layer connected in sequence. The decoder second module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The decoder third module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The decoder fourth module includes a ResBlock block, an upsampling layer, and a ResBlock block connected in sequence. The time step embedding vector generation module includes a linear layer, a SiLU activation function, and a linear layer connected in sequence; the encoder includes a first encoder module, a second encoder module, a third encoder module, and a fourth encoder module connected in sequence, the first encoder module includes a ResBlock block and a downsampling layer connected in sequence, the second encoder module includes a ResBlock block and a downsampling layer connected in sequence, the third encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence, and the fourth encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence. Specifically, the ResBlock block refers to a 3x3 ResBlock, the downsampling layer refers to a 2x downsampling layer, and the upsampling layer refers to a 2x upsampling layer.

[0112] In some embodiments, the Wavelet block includes a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence. The discrete wavelet transform module is used to convert linear structure information into frequency features, the ResBlock block is used to separate Gaussian noise according to the frequency of the frequency features, and the inverse discrete wavelet transform module is used to convert the frequency features into linear structure information.

[0113] It can be seen that the first clean image in the face data set is obtained; the first clean image is input into the diffusion process of the diffusion model, and Gaussian noise is gradually added in the diffusion process to obtain a noisy image; the diffusion model includes a time step embedding vector generation module, an encoder, and a decoder. The decoder includes a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence. The decoder first module includes a Wavelet block, an Attention block, a Wavelet block, and an upsampling layer connected in sequence. The decoder second module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The decoder third module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The decoder fourth module includes a ResBlock block, an upsampling layer, and a ResBlock block connected in sequence. The Wavelet block is used to extract the first frequency feature of the first feature map through a discrete wavelet transform module; a loss function of the diffusion model is constructed based on the facial features, the noise image is input into the reconstruction process of the diffusion model, and the diffusion model is trained in the reconstruction process based on the loss function to obtain a target diffusion model; a random Gaussian noise image is obtained, and the random Gaussian noise image is input into the reconstruction process of the target diffusion model to obtain a generated face image. By introducing the Wavelet block into the diffusion model, the Wavelet block can be used to extract the first frequency feature of the first feature map, that is, to extract the high-frequency and low-frequency information of the noise image. The diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, and can effectively separate Gaussian noise. The Wavelet block has a high accuracy in extracting the frequency information of the noise image, which can improve the accuracy of separating Gaussian noise, improve the overall quality of the generated face image, and make the generated face image more realistic and clear. In addition, a loss function of the diffusion model is constructed based on the facial features, and the diffusion model is trained based on the loss function during the reconstruction process, guiding the diffusion model to pay more attention to the details of the facial features, thereby improving the quality of the facial details of the generated face image.

[0114] In some embodiments, a first clean image is obtained from a face data set as a training set, and the first clean image in the training set is input into the diffusion process of a diffusion model, and Gaussian noise is gradually added during the diffusion process until the first clean image is completely covered by Gaussian noise to obtain a noisy image. An additional MSE loss is constructed based on the prior information of facial features, and the mean square error loss MSE and the additional MSE loss are combined to form a loss function for training the diffusion network of the diffusion model. The noisy image is input into the reconstruction process of the diffusion model, and the diffusion network is trained based on the loss function during the reconstruction process to obtain a target diffusion network; a random Gaussian noise image is obtained, and the random Gaussian noise image is input into the reconstruction process of the diffusion model, and the target diffusion network is used to obtain a generated face image. Among them, inputting the noisy image into the reconstruction process of the diffusion model and training the diffusion network based on the loss function during the reconstruction process includes: inputting the noisy image into the reconstruction process of the diffusion model, and using the diffusion network to gradually infer the Gaussian noise added to the noisy image based on the loss function during the reconstruction process, and gradually removing the added Gaussian noise to obtain a second clean image. Among them, constructing an additional MSE loss based on the prior information of facial features includes: extracting features of the first clean image to obtain facial features and non-facial features; dilating the facial features to cover all areas within a preset range around the facial features, and dilating the non-facial features to cover the edge areas; combining the facial features and non-facial features after the dilation operation into a mask image; blurring the mask image to obtain a target mask image; and obtaining an additional MSE loss based on the target mask image.

[0115] Among them, the diffusion network includes a time step embedding vector generation module, an encoder, and a decoder. The decoder includes a first decoder module, a second decoder module, a third decoder module, and a fourth decoder module connected in sequence. The first decoder module includes a Wavelet block, an Attention block, a Wavelet block, and an upsampling layer connected in sequence. The second decoder module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The third decoder module includes a ResBlock block, an Attention block, and an upsampling layer connected in sequence. The fourth decoder module includes a ResBlock block, an upsampling layer, and a ResBlock block connected in sequence. The Wavelet block is used to extract the first frequency feature of the first feature map through a discrete wavelet transform module. The time step embedding vector generation module includes a linear layer, a SiLU activation function, and a linear layer connected in sequence; the encoder includes a first encoder module, a second encoder module, a third encoder module, and a fourth encoder module connected in sequence, the first encoder module includes a ResBlock block and a downsampling layer connected in sequence, the second encoder module includes a ResBlock block and a downsampling layer connected in sequence, the third encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence, and the fourth encoder module includes a ResBlock block, an Attention block, and a downsampling layer connected in sequence. The Wavelet block includes a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map, the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain the second frequency feature, and the inverse discrete wavelet transform module is used to convert the second frequency feature into a second feature map.

[0116] The disclosed embodiment also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image generation method based on the diffusion model when executing the computer program. The electronic device can be any intelligent terminal including a computer.

[0117] The embodiment of the present disclosure further provides a storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned image generation method based on the diffusion model is implemented.

[0118] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0119] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0120] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0121] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0122] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. An image generation method based on a diffusion model, characterized in that: include: Get the first clean image in the face dataset; Inputting the first clean image into a diffusion process of a diffusion model, and gradually adding Gaussian noise during the diffusion process to obtain a noisy image; The diffusion model includes a time step embedding vector generation module, an encoder, and a decoder. The decoder includes a first decoder module, a second decoder module, a third decoder module, and a fourth decoder module connected in sequence. The first decoder module includes a Wavelet block, an Attention block, the Wavelet block, and an upsampling layer connected in sequence. The second decoder module includes a ResBlock block, the Attention block, and the upsampling layer connected in sequence. The third decoder module includes the ResBlock block, the Attention block, and the upsampling layer connected in sequence. The fourth decoder module includes the ResBlock block, the upsampling layer, and the ResBlock block connected in sequence. The Wavelet block is used to extract a first frequency feature of a first feature map through a discrete wavelet transform module. Constructing the loss function of the diffusion model based on facial features; Inputting the noise image into a reconstruction process of the diffusion model, training the diffusion model in the reconstruction process based on the loss function to obtain a target diffusion model; A random Gaussian noise image is acquired, and the random Gaussian noise image is input into the reconstruction process of the target diffusion model to obtain a generated face image.

2. The image generation method based on the diffusion model according to claim 1, characterized in that: The step of inputting the first clean image into a diffusion process of a diffusion model and gradually adding Gaussian noise in the diffusion process to obtain a noisy image includes: The first clean image is input into the diffusion process of the diffusion model, and the Gaussian noise is gradually added through a diffusion formula in the diffusion process to obtain the noisy image. The diffusion formula is: In the formula, α t ∈(0,1), x0 is the first clean image, x t is the noise image after adding the Gaussian noise for t steps, ε is the Gaussian noise, and t∈(1,T).

3. The image generation method based on the diffusion model according to claim 1, characterized in that: The step of inputting the noise image into the reconstruction process of the diffusion model and training the diffusion model based on the loss function during the reconstruction process includes: The noisy image is input into the reconstruction process of the diffusion model. Based on the loss function, the Gaussian noise added to the noisy image is gradually inferred through a reconstruction formula during the reconstruction process, and the added Gaussian noise is gradually removed to obtain a second clean image. The reconstruction formula is: In the formula, x t is the noise image after adding the Gaussian noise after t steps, x t-1 is the noise image after adding the Gaussian noise in step t-1, α t ∈(0,1), t∈(1,T),ε θ (x t ,t) is the Gaussian noise added under the inferred parameter θ.

4. The image generation method based on a diffusion model according to claim 1, characterized in that: The loss function of the diffusion model constructed based on the facial features includes: Constructing an additional MSE loss based on the facial features; The loss function of the diffusion model is composed of the mean square error loss MSE and the additional MSE loss. The loss function of the diffusion model is: Where, L is the loss function of the diffusion model, L MSE is the mean square error loss MSE, L attribute is the additional MSE loss.

5. The image generation method based on the diffusion model according to claim 4, characterized in that: The additional MSE loss is constructed based on the facial features, including: Performing feature extraction on the first clean image to obtain the facial features and non-facial features of the human face; Performing an expansion operation on the facial features to cover all areas within a preset range around the facial features, and performing the expansion operation on the non-facial features to cover edge areas; Combining the facial features and the non-facial features after the dilation operation into a mask image; Performing a blur operation on the mask image to obtain a target mask image; According to the target mask image, the additional MSE loss is obtained: Where x0 is the first clean image, x t is the noise image after adding the Gaussian noise after t steps, ε is the Gaussian noise, ε θ (x t ,t) is the added Gaussian noise inferred under the θ parameter, M is the target mask image, t∈(1,T).

6. The image generation method based on the diffusion model according to claim 4, characterized in that: The mean square error loss MSE is: Where x0 is the first clean image, x t is the noise image after adding the Gaussian noise after t steps, ε is the Gaussian noise, ε θ (x t ,t) is the inferred Gaussian noise added under the θ parameter.

7. The image generation method based on diffusion model according to claim 1, characterized in that: The Wavelet block includes the discrete wavelet transform module, the ResBlock block, and the inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map, the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain the second frequency feature, and the inverse discrete wavelet transform module is used to convert the second frequency feature into a second feature map.

8. The image generation method based on diffusion model according to claim 1, characterized in that: The time step embedding vector generation module includes a linear layer, a SiLU activation function, and the linear layer connected in sequence; the encoder includes a first encoder module, a second encoder module, a third encoder module, and a fourth encoder module connected in sequence, the first encoder module includes the ResBlock block and the downsampling layer connected in sequence, the second encoder module includes the ResBlock block and the downsampling layer connected in sequence, the third encoder module includes the ResBlock block, the Attention block, and the downsampling layer connected in sequence, and the fourth encoder module includes the ResBlock block, the Attention block, and the downsampling layer connected in sequence.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the image generation method based on a diffusion model as described in any one of claims 1 to 8 when executing the computer program.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image generation method based on a diffusion model according to any one of claims 1 to 8 is implemented.

Images