An image generation method based on a diffusion model

Abstract

The embodiment of the disclosure discloses a kind of image generation methods based on diffusion model, the diffusion process of first clean image input diffusion model is obtained, and noise image is obtained, diffusion model includes Wavelet block and is based on facial feature of five organs Loss function of diffusion model is constructed, noise image is input into the reconstruction process of diffusion model, diffusion model is trained, and target diffusion model is obtained, random Gaussian noise image is obtained, and random Gaussian noise image is input into the reconstruction process of target diffusion model, and the generated face image is obtained.In diffusion model, Wavelet block is introduced, Wavelet block can be used to extract the first frequency feature of first feature map, that is, the high-frequency and low-frequency information of noise image is extracted, and the diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, which can effectively separate Gaussian noise, improve the quality of the generated face image, and construct the loss function based on the facial feature of five organs, improve the quality of the facial feature of the generated face image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing and computer vision, and particularly relates to a diffusion model-based image generation method. BACKGROUND

[0002] In recent years, face image generation technology has developed rapidly. Currently, typical models for face image generation include GAN, VAE and diffusion model. GAN has a wide range of applications and can generate high-fidelity and diversified face images, and has great flexibility. However, the training process of GAN is unstable and prone to mode collapse. The training process of VAE is stable and not prone to mode collapse. However, the face images generated by VAE have low fidelity and blurred details, and in most cases, the generated face images have a strong synthetic sense. The training process of the diffusion model is stable, and the generated face images are high-fidelity, diversified and flexible. However, the quality of the face images generated by the diffusion model is low, and the details are blurred. SUMMARY

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of protection of the claims.

[0004] The embodiment of the present disclosure provides a diffusion model-based image generation method, which can improve the overall quality of the generated face images and make the generated face images more realistic and clear.

[0005] In an embodiment of the first aspect of the present disclosure, a diffusion model-based image generation method comprises:

[0006] obtaining a first clean image in a face data set;

[0007] inputting the first clean image into a diffusion process of a diffusion model, gradually adding Gaussian noise in the diffusion process to obtain a noise image; the diffusion model comprises a time step embedding vector generation module, an encoder and a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, the Wavelet block and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, the Attention block and the up-sampling layer connected in sequence, the decoder third module comprises the ResBlock block, the Attention block and the up-sampling layer connected in sequence, and the decoder fourth module comprises the ResBlock block, the up-sampling layer and the ResBlock block connected in sequence, and the Wavelet block is configured to extract first frequency features of a first feature map through a discrete wavelet transform module;

[0008] constructing a loss function of the diffusion model based on facial feature characteristics;

[0009] inputting the noise image into a reconstruction process of the diffusion model, training the diffusion model in the reconstruction process based on the loss function, and obtaining a target diffusion model;

[0010] obtaining a random Gaussian noise image, inputting the random Gaussian noise image into the reconstruction process of the target diffusion model, and obtaining a generated facial image.

[0011] According to certain embodiments of the first aspect of the present disclosure, the inputting the first clean image into a diffusion process of a diffusion model and gradually adding Gaussian noise in the diffusion process to obtain a noise image comprises:

[0012] inputting the first clean image into the diffusion process of the diffusion model and gradually adding the Gaussian noise in the diffusion process by a diffusion formula to obtain the noise image, the diffusion formula being:

[0013]

[0014] wherein, α t ∈(0, 1), x0is the first clean image, x t is the noise image after adding the Gaussian noise for t steps, and ε is the Gaussian noise, t ∈(1, T).

[0015] According to certain embodiments of the first aspect of the present disclosure, the inputting the noise image into a reconstruction process of the diffusion model and training the diffusion model in the reconstruction process based on the loss function comprises:

[0016] inputting the noise image into the reconstruction process of the diffusion model, and based on the loss function, gradually inferring the added Gaussian noise in the noise image in the reconstruction process by a reconstruction formula and gradually removing the added Gaussian noise to obtain a second clean image, the reconstruction formula being:

[0017]

[0018] wherein, x t is the noise image after adding the Gaussian noise for t steps, x t-1 is the noise image after adding the Gaussian noise for t-1 steps, α t ∈(0, 1), t ∈(1, T), and ε θ (x t , t) is the inferred added Gaussian noise under the θ parameter.

[0019] According to some embodiments of the first aspect of the present disclosure, the loss function of the diffusion model is constructed based on the facial feature, and the loss function of the diffusion model comprises:

[0020] An additional MSE loss is constructed based on the facial feature;

[0021] The loss function of the diffusion model is composed of a mean square error loss MSE and the additional MSE loss, and the loss function of the diffusion model is:

[0022]

[0023] In the formula, L is the loss function of the diffusion model, L MSE is the mean square error loss MSE, and L attribute is the additional MSE loss.

[0024] According to some embodiments of the first aspect of the present disclosure, the additional MSE loss is constructed based on the facial feature, and the additional MSE loss comprises:

[0025] Feature extraction is performed on the first clean image to obtain the facial feature and a non-facial feature;

[0026] An inflation operation is performed on the facial feature to cover all regions within a preset range around the facial feature, and the inflation operation is performed on the non-facial feature to cover an edge region;

[0027] The facial feature and the non-facial feature after the inflation operation are combined into a mask picture;

[0028] A blur operation is performed on the mask picture to obtain a target mask picture;

[0029] The additional MSE loss is obtained according to the target mask picture:

[0030]

[0031] In the formula, x0 is the first clean image, x t is the noise image after adding the Gaussian noise for t steps, ε is the Gaussian noise, ε θ (x t , t) is the Gaussian noise added under the θ parameter, M is the target mask picture, and t ∈ (1, T).

[0032] According to some embodiments of the first aspect of the present disclosure, the mean square error loss MSE is:

[0033]

[0034] wherein x0 is the first clean image, x t is the noisy image after adding the Gaussian noise for t steps, and ε is the Gaussian noise, ε θ (x t is the estimated added Gaussian noise under the θ parameter.

[0035] According to some embodiments of the first aspect of the present disclosure, the Wavelet block comprises the discrete wavelet transform module, the ResBlock block, and an inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is configured to extract the first frequency feature of the first feature map; the ResBlock block is configured to separate the Gaussian noise according to the high and low frequencies of the first frequency feature to obtain a second frequency feature; and the inverse discrete wavelet transform module is configured to convert the second frequency feature into a second feature map.

[0036] According to some embodiments of the first aspect of the present disclosure, the time step embedding vector generation module comprises a linear layer, a SiLU activation function, and another linear layer connected in sequence; and the encoder comprises an encoder first module, an encoder second module, an encoder third module, and an encoder fourth module connected in sequence; the encoder first module comprises the ResBlock block and a down-sampling layer connected in sequence; the encoder second module comprises the ResBlock block and the down-sampling layer connected in sequence; the encoder third module comprises the ResBlock block, the Attention block, and the down-sampling layer connected in sequence; and the encoder fourth module comprises the ResBlock block, the Attention block, and the down-sampling layer connected in sequence.

[0037] According to an embodiment of the second aspect of the present disclosure, an electronic device comprises a memory and a processor; the memory stores a computer program; and the processor executes the computer program to implement the image generation method based on the diffusion model.

[0038] According to an embodiment of the third aspect of the present disclosure, a storage medium stores a computer program; and the computer program is executed by a processor to implement the image generation method based on the diffusion model.

[0039] The embodiments of the present disclosure at least have the following beneficial effects: a first clean image in a face data set is obtained; the first clean image is input into a diffusion process of a diffusion model, and Gaussian noise is gradually added in the diffusion process to obtain a noisy image; the diffusion model comprises a time step embedding vector generation module, an encoder, and a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, a Wavelet block, and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, the decoder third module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, and the decoder fourth module comprises a ResBlock block, an up-sampling layer, and a ResBlock block connected in sequence, the Wavelet block is used to extract first frequency features of a first feature map through a discrete wavelet transform module; a loss function of the diffusion model is constructed based on facial feature characteristics, the noisy image is input into a reconstruction process of the diffusion model, the diffusion model is trained in the reconstruction process based on the loss function to obtain a target diffusion model; and a random Gaussian noise image is obtained, and the random Gaussian noise image is input into the reconstruction process of the target diffusion model to obtain a generated face image. In the diffusion model, the Wavelet block is introduced, the Wavelet block can be used to extract the first frequency features of the first feature map, that is, high-frequency and low-frequency information of the noisy image, the diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, can effectively separate the Gaussian noise, and the accuracy of the Wavelet block in extracting the frequency information of the noisy image is relatively high, which can improve the accuracy of separating the Gaussian noise and improve the overall quality of the generated face image, making the generated face image more realistic and clear. In addition, the loss function of the diffusion model is constructed based on facial feature characteristics, the diffusion model is trained in the reconstruction process based on the loss function, and the diffusion model is guided to pay more attention to the details of the facial feature characteristics, thereby improving the quality of the facial feature characteristics of the generated face image.

[0040] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings are included to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0042] Figure 1 An optional flowchart of an image generation method based on a diffusion model provided by the embodiments of the present disclosure;

[0043] Figure 2 A diffusion network structure schematic diagram provided by an embodiment of the present disclosure;

[0044] Figure 3 A diffusion process structure schematic diagram provided by an embodiment of the present disclosure;

[0045] Figure 4 A reconstruction process structure schematic diagram provided by an embodiment of the present disclosure;

[0046] Figure 5 A Wavelet block structure schematic diagram provided by an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram of a generated face image obtained from a random Gaussian noise image provided by an embodiment of the present disclosure;

[0048] Figure 7 A schematic diagram of a target mask picture obtained based on facial feature provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and not intended to limit the present disclosure.

[0050] In an embodiment of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0052] In order to facilitate understanding of the technical solutions provided by the embodiments of the present disclosure, some key terms used by the embodiments of the present disclosure are explained first:

[0053] Mean Squared Error (MSE) is a commonly used loss function, which is often used to evaluate the difference between the predicted value and the true value of the model. It measures the prediction performance of the model by calculating the average of the error squares between the predicted value and the true value. The smaller the mean squared error, the more accurate the model's prediction; the larger the mean squared error, the larger the model's prediction error.

[0054] Discrete wavelet transform (DWT) is a discrete scale and translation of the basic wavelet, which can decompose signals or images into components of different frequencies.

[0055] Inverse discrete wavelet transform (IDWT) is the inverse process of discrete wavelet transform, which combines the frequency bands decomposed by discrete wavelet transform into the original signal.

[0056] The embodiments of the present disclosure are further described below with reference to the accompanying drawings.

[0057] Referring to Figure 1 As shown in Figure 1 An optional flowchart of an image generation method based on a diffusion model provided by the present disclosure, which includes but is not limited to steps 101 to 105. Each step will be introduced in turn.

[0058] Step 101: Obtain a first clean image in a face data set.

[0059] Specifically, the face data set can be the CelebAMask-HQ data set, the Flickr-Faces-Hight-Quality data set, or the Landmark guided face Parsing data set. The selection of the face data set can be set according to the actual situation, and the present embodiment does not make specific limitations thereto. The first clean image refers to a clear, high-quality face image that has not been contaminated or disturbed by noise.

[0060] Step 102: input the first clean image into the diffusion process of the diffusion model, and gradually add Gaussian noise in the diffusion process to obtain a noisy image.

[0061] Specifically, the diffusion process of the diffusion model refers to gradually adding noise to the original data to convert the original data into noise data.

[0062] In some embodiments, inputting the first clean image into the diffusion process of the diffusion model, and gradually adding Gaussian noise in the diffusion process to obtain a noisy image, comprises:

[0063] The first clean image is input into the diffusion process of the diffusion model. During the diffusion process, Gaussian noise is gradually added using the diffusion formula to obtain a noisy image. The diffusion formula is:

[0064]

[0065] In the formula, α t ∈(0,1), x0 is the first clean image, x t Let α be the noisy image after adding Gaussian noise for t steps, where ε is the Gaussian noise and t∈(1,T). t The coefficient is used to control the Gaussian noise intensity. N is the standard Gaussian distribution, and I is the variance.

[0066] Reference Figure 3 As shown, Figure 3 This is a schematic diagram of the diffusion process structure provided in an embodiment of this disclosure. A first clean image x0 is input into the diffusion process of the diffusion model. Gaussian noise is added to the first clean image x0 during the diffusion process. The diffusion process has a total of T steps. Gaussian noise is added at each step of the diffusion process, where Gaussian noise is added according to the diffusion formula at each step. The noisy image obtained after adding Gaussian noise at step t is x. t .

[0067] Specifically, Gaussian noise refers to a type of noise whose probability density function follows a normal distribution. It is understandable that Gaussian noise is a type of noise widely present in nature, and image data in real-world scenarios is mostly affected by Gaussian noise. Adding Gaussian noise to the first clean image can more accurately simulate the impact on image data in real-world scenarios, improving the robustness of the diffusion model and enabling it to better adapt to and handle changes and noise in the real world.

[0068] Step 103: Construct the loss function of the diffusion model based on facial features.

[0069] Specifically, the loss function is an important concept in machine learning and deep learning, used to measure the difference or error between the model's predictions and the true results. It is a numerical evaluation metric that provides a measure of model performance by comparing the model's output with the true labels. When training a diffusion model, adjusting the model's parameters to minimize the loss function improves the model's performance.

[0070] In some embodiments, the loss function of the diffusion model is constructed based on facial features, including: constructing an additional MSE loss based on facial features; combining the mean squared error loss (MSE) and the additional MSE loss to form the loss function of the diffusion model, wherein the loss function of the diffusion model is:

[0071]

[0072] In the formula, L is a loss function of the diffusion model, L MSE is a mean square error loss MSE, L attribute is an additional MSE loss. λ is a weight parameter.

[0073] It can be understood that, on the basis of using the mean square error loss MSE, the additional MSE loss based on the facial feature is added to obtain the loss function of the diffusion model, and the diffusion model is trained based on the loss function in the reconstruction process, so as to guide the diffusion model to pay more attention to the details of the facial feature region and improve the quality of the facial feature details of the generated facial image.

[0074] Referring to Figure 7 , a schematic diagram of obtaining a target mask picture based on a facial feature is provided in the embodiments of the present disclosure. In some embodiments, the additional MSE loss is constructed based on the facial feature, including: Figure 7

[0075] The facial feature and the non-facial feature are obtained by performing feature extraction on the first clean image;

[0076] The facial feature is dilated to cover all regions within a preset range around the facial feature, and the non-facial feature is dilated to cover the edge region;

[0077] The facial feature and the non-facial feature after the dilating operation are combined into a mask picture;

[0078] The mask picture is blurred to obtain a target mask picture;

[0079] According to the target mask picture, the additional MSE loss is obtained:

[0080]

[0081] In the formula, x0 is the first clean image, x t is a noise image after adding Gaussian noise for t steps, ε is Gaussian noise, ε θ (x t , t) is the predicted added Gaussian noise under the θ parameter, M is the target mask picture, t ∈ (1, T). E is an expected value.

[0082] ​Specifically, the facial feature refers to elements such as eyes, nose, mouth, and ears that constitute key parts of a face, and the non-facial feature refers to features of the first clean image extracted other than the facial feature. The dilation operation refers to a morphological operation of expanding the boundary of an object in image processing, and the blurring operation refers to a technique for reducing image noise and details to make the image look softer and smoother in image processing. The mask picture refers to a face picture formed by reassembling the facial feature and the non-facial feature after the dilation operation.

[0083] It can be understood that the facial feature and the non-facial feature are obtained by performing feature extraction on the first clean image, the facial feature is subjected to the dilation operation to cover all areas within a preset range around the facial feature to emphasize all relevant information within the preset range around the facial feature, and the non-facial feature is subjected to the dilation operation to cover the edge region of the non-facial feature, so that the edge information is considered in the loss calculation to improve the retention and reconstruction capability of the diffusion model for the image edge. The preset range can be set according to actual conditions, and the present embodiment does not specifically limit it.

[0084] It can be understood that the facial feature and the non-facial feature after the dilation operation are combined into a mask picture, and the mask picture is subjected to the blurring operation to make the edge smoother and the weight distribution smoother, and reduce the possibility of sudden changes or discontinuity.

[0085] It can be understood that after obtaining the target mask picture, the additional MSE loss is obtained according to the target mask picture, and the loss function of the diffusion model is composed of the mean square error loss MSE and the additional MSE loss. By adding the target mask picture to the loss function of the diffusion model, the weight of the facial feature is larger during the training of the diffusion model, which guides the diffusion model to pay more attention to the details of the facial feature region, so that the diffusion model is trained in the direction of smaller facial feature loss, and the facial details and features are better preserved, and the quality of the generated facial image is improved.

[0086] In some embodiments, the facial feature and the non-facial feature are obtained by performing feature extraction on the first clean image, and the facial feature includes an eye feature, the eye feature is subjected to the dilation operation to cover all areas within a preset range around the eye feature, and the preset range can be a range of shape and texture information containing the eye socket to emphasize the shape and texture information of the eye socket around the eye.

[0087] In some embodiments, the mean square error loss MSE is:

[0088]

[0089] In the formula, x0 is the first clean image, x t The image is a noisy image after t steps of adding Gaussian noise, where ε is the Gaussian noise. θ (x t ,t) represents the predicted Gaussian noise added under the parameter θ. E is the expected value.

[0090] Specifically, the mean squared error loss (MSE) provides a quantitative measure of the accuracy of diffusion model predictions by calculating the average of the squared differences between predicted and true values. This measurement method is simple and intuitive. The formula for calculating the MSE is simple and clear, easy to implement, and computationally efficient, thus improving the efficiency of training diffusion models.

[0091] Step 104: Input the noisy image into the reconstruction process of the diffusion model, and train the diffusion model based on the loss function during the reconstruction process to obtain the target diffusion model.

[0092] Specifically, the reconstruction process of the diffusion model involves gradually denoising the noisy data to restore it to its original state.

[0093] In some embodiments, a noisy image is input into the reconstruction process of a diffusion model, and the diffusion model is trained during the reconstruction process based on a loss function, including:

[0094] The noisy image is input into the reconstruction process of the diffusion model. Based on the loss function, the Gaussian noise added to the noisy image is gradually inferred through the reconstruction formula during the reconstruction process, and the added Gaussian noise is gradually removed to obtain a second clean image. The reconstruction formula is as follows:

[0095]

[0096] In the formula, σ t x is a constant. t For the noisy image after adding Gaussian noise in t steps, x t-1 For the noisy image after adding Gaussian noise in step t-1, α t ∈(0,1), t∈(1,T), ε θ (x t ,t) represents the Gaussian noise inferred under the parameter θ. θ represents the parameters of the neural network to be trained, i.e., the parameters that the diffusion network of the diffusion model needs to be trained. ε represents the Gaussian noise, and α t The coefficient is used to control the Gaussian noise intensity. N is the standard Gaussian distribution, and I is the variance.

[0097] Reference Figure 4 As shown, Figure 4 A schematic diagram of the reconstruction process provided in an embodiment of this disclosure. tx t is a noisy image to which Gaussian noise is added at step t t-1 x t-1 is a noisy image to which Gaussian noise is added at step t-1 θ (x t t) is the Gaussian noise added at step t inferred under the θ parameter θ (x t-1 t-1) is the Gaussian noise added at step t-1 inferred under the θ parameter.

[0098] Specifically, the first clean image is input into the diffusion process of the diffusion model, Gaussian noise is added step by step through the diffusion formula in the diffusion process to obtain a noisy image, the noisy image is input into the reconstruction process of the diffusion model, the added Gaussian noise in the noisy image is inferred step by step through the reconstruction formula in the reconstruction process, and the added Gaussian noise is removed step by step to obtain a second clean image. The second clean image obtained after denoising is the first clean image before adding noise. It can be understood that the diffusion model can improve the quality of the face image generated by the diffusion model by adding Gaussian noise to the first clean image step by step and then learning how to reverse this process to generate high-quality data. And by adding Gaussian noise and training the diffusion model to remove Gaussian noise, the robustness of the diffusion model to input data is enhanced, so that the diffusion model can perform more stably when processing input data with Gaussian noise.

[0099] Step 105: Obtain a random Gaussian noise image, input the random Gaussian noise image into the reconstruction process of the target diffusion model, and obtain a generated face image.

[0100] Specifically, the random Gaussian noise image refers to a face image covered by Gaussian noise of different intensities. It can be understood that the random Gaussian noise image is input into the reconstruction process of the target diffusion model, the Gaussian noise in the random Gaussian noise image is inferred step by step in the reconstruction process, and the Gaussian noise is removed step by step to obtain a generated face image.

[0101] Referring to Figure 6 , Figure 6 is a schematic diagram of obtaining a generated face image from a random Gaussian noise image provided by the embodiments of the present disclosure, and Figure 6 the random Gaussian noise image in the left image of Figure 6 is input into the reconstruction process of the target diffusion model to obtain Figure 6 the generated face image in the right image of Figure 6 .

[0102] Specifically, the diffusion model comprises a time step embedding vector generation module, an encoder, a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, a Wavelet block and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, an Attention block and an up-sampling layer connected in sequence, the decoder third module comprises a ResBlock block, an Attention block and an up-sampling layer connected in sequence, and the decoder fourth module comprises a ResBlock block, an up-sampling layer and a ResBlock block connected in sequence, and the Wavelet block is used for extracting first frequency features of the first feature map through a discrete wavelet transform module.

[0103] In some embodiments, the time step embedding vector generation module comprises a linear layer, a SiLU activation function and a linear layer connected in sequence; and the encoder comprises an encoder first module, an encoder second module, an encoder third module and an encoder fourth module connected in sequence, the encoder first module comprises a ResBlock block and a down-sampling layer connected in sequence, the encoder second module comprises a ResBlock block and a down-sampling layer connected in sequence, the encoder third module comprises a ResBlock block, an Attention block and a down-sampling layer connected in sequence, and the encoder fourth module comprises a ResBlock block, an Attention block and a down-sampling layer connected in sequence.

[0104] The ResBlock block refers to a residual block. The SiLU (Sigmoid Linear Unit) activation function is an adaptive activation function. The time step embedding vector generation module is used to generate a time step embedding vector according to the current time. The Attention block is used to weight the input feature map. The linear layer is also called a fully connected layer, which is a layer used to realize linear transformation in a neural network. The up-sampling layer is a technique used in deep learning to increase the resolution or size of data. The down-sampling layer, also known as a pooling layer, is used to reduce the dimension and computation of data while retaining the main features. Specifically, the encoder is responsible for compressing the input noise image into a low-dimensional representation while extracting key features; the decoder is responsible for reconstructing the original noise image size from the low-dimensional representation output by the encoder and separating the Gaussian noise in the process; the time step embedding vector generation module is responsible for encoding the time step information into a vector form as a supplementary feature of the original sequence data, enhancing the understanding and prediction ability of the diffusion model for time series data. The time step embedding vector generation module takes the current time t as input and outputs a time step embedding vector, which is input to the encoder together with the noise image as input data. After the noise image is processed by the encoder, it is input to the decoder together with the time step embedding vector as input data. The decoder separates the Gaussian noise and outputs the second clean image obtained after denoising. Through the cooperative work of the time step embedding vector generation module, the encoder and the decoder, the Gaussian noise in the noise image is effectively predicted and removed.

[0105] In the encoder, the ResBlock block adjusts the extraction of noise information according to different time steps by learning the input feature map and time step embedding vector, helping the diffusion model to understand time series data. The skip connection structure of the ResBlock block makes the diffusion model training process more stable; the Attention block weights the input feature map, making the diffusion model pay more attention to the facial feature area of the noise image, and can also highlight the noise area in the noise image, more effectively removing the Gaussian noise. The down-sampling layer reduces the dimension of the data by reducing the size of the feature map while retaining important information.

[0106] In the decoder, the up-sampling layer is used to enlarge the size of the feature map. The first module of the decoder contains four blocks, the first block is the Wavelet block, which is used to extract the first frequency feature of the first feature map through the discrete wavelet transform module. The third block is the Wavelet block, which is used to convert the processed first frequency feature back to the spatial domain. The two Wavelet blocks refine the features of the first feature map and effectively separate the Gaussian noise, and the second module of the decoder and the third module of the decoder further refine the features of the feature map, improve the image quality, and the fourth module of the decoder finely adjusts and integrates the features of the feature map.

[0107] In some embodiments, referring to Figure 5 , the Wavelet block structure diagram provided by the embodiments of the present disclosure is shown. Figure 5 The Wavelet block includes a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map, the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain the second frequency feature, and the inverse discrete wavelet transform module is used to convert the second frequency feature into the second feature map.

[0108] Specifically, the first feature map, the second feature map, and the feature map refer to a series of intermediate representations generated in the process of processing the noisy image by the encoder or the decoder to generate the final output, i.e., the second clean image, and the first feature map, the second feature map, and the feature map contain the key information of the noisy image and the features learned by the encoder or the decoder in the process of gradually removing the Gaussian noise and recovering the details of the second clean image. The first frequency feature refers to the feature in a different frequency range extracted after the noisy image is divided into different frequency bands by discrete wavelet transform, and the second frequency feature refers to the feature after the first frequency feature is separated from the Gaussian noise.

[0109] In the decoder, the Wavelet block extracts the first frequency feature of the first feature map through the discrete wavelet transform module, separates the Gaussian noise according to the frequency of the first frequency feature through the ResBlock block to obtain the second frequency feature, and converts the second frequency feature into the second feature map through the inverse discrete wavelet transform module. Unlike the ResBlock block of the encoder, the ResBlock block of the decoder also needs to receive a feature map of the same size from the ResBlock block in the encoder as input, that is, the encoder and the decoder directly pass the input to the output through the ResBlock block of the skip connection, which helps to solve the gradient disappearance problem and improves the training stability of the diffusion model.

[0110] It can be understood that the preset frequency range is set to separate the Gaussian noise according to the frequency range in which the Gaussian noise added in the noise image exists, after the discrete wavelet transform module extracts the first frequency feature of the first feature map, the ResBlock block separates the Gaussian noise of the first frequency feature according to the preset frequency range, obtains the second frequency feature, and the inverse discrete wavelet transform module converts the second frequency feature into the second feature map. The preset frequency range can be set according to actual conditions, and the embodiment does not make specific limitations thereto. Specifically, the Wavelet block is introduced into the diffusion model, the first frequency feature of the first feature map can be extracted using the Wavelet block, that is, the frequency information of the noise image is extracted, the ResBlock block in the Wavelet block separates the Gaussian noise according to the frequency information and the preset frequency range, which can effectively separate the Gaussian noise, and the accuracy of the Wavelet block extracting the frequency information of the noise image is relatively high, which can improve the accuracy of separating the Gaussian noise, improve the overall quality of the generated face image, and make the generated face image more realistic and clear.

[0111] wherein the model refers to the combination of network structure and parameters, the diffusion model includes a diffusion network, referring to Figure 2 Figure 2 A diffusion network structure diagram provided by the embodiment of the present disclosure. The diffusion network includes a time step embedding vector generation module, an encoder, and a decoder, the decoder includes a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module includes a Wavelet block, an Attention block, a Wavelet block, and an up-sampling layer connected in sequence, the decoder second module includes a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, the decoder third module includes a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, and the decoder fourth module includes a ResBlock block, an up-sampling layer, and a ResBlock block connected in sequence. The time step embedding vector generation module includes a linear layer, a SiLU activation function, and a linear layer connected in sequence; the encoder includes an encoder first module, an encoder second module, an encoder third module, and an encoder fourth module connected in sequence, the encoder first module includes a ResBlock block and a down-sampling layer connected in sequence, the encoder second module includes a ResBlock block and a down-sampling layer connected in sequence, the encoder third module includes a ResBlock block, an Attention block, and a down-sampling layer connected in sequence, and the encoder fourth module includes a ResBlock block, an Attention block, and a down-sampling layer connected in sequence. Specifically, the ResBlock block refers to a 3x3 ResBlock, the down-sampling layer refers to a 2x down-sampling layer, and the up-sampling layer refers to a 2x up-sampling layer.

[0112] ​In some embodiments, the Wavelet block comprises a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence, the discrete wavelet transform module is configured to convert the linear structure information into frequency features, the ResBlock block is configured to separate Gaussian noise according to the frequency of the frequency features, and the inverse discrete wavelet transform module is configured to convert the frequency features into linear structure information.

[0113] It can be seen that a first clean image in the face data set is obtained; the first clean image is input into a diffusion process of a diffusion model, and Gaussian noise is gradually added in the diffusion process to obtain a noisy image; the diffusion model comprises a time step embedding vector generation module, an encoder, and a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, a Wavelet block, and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, the decoder third module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, and the decoder fourth module comprises a ResBlock block, an up-sampling layer, and a ResBlock block connected in sequence, the Wavelet block is configured to extract first frequency features of a first feature map through a discrete wavelet transform module; a loss function of the diffusion model is constructed based on facial feature characteristics, the noisy image is input into a reconstruction process of the diffusion model, the diffusion model is trained in the reconstruction process based on the loss function to obtain a target diffusion model, and a generated face image is obtained by inputting a random Gaussian noise image into the reconstruction process of the target diffusion model. In the diffusion model, the Wavelet block is introduced to extract the first frequency features of the first feature map, that is, to extract high-frequency and low-frequency information of the noisy image, the diffusion model separates Gaussian noise according to the high-frequency and low-frequency information, can effectively separate Gaussian noise, and the accuracy of the Wavelet block in extracting frequency information of the noisy image is high, which can improve the accuracy of separating Gaussian noise and improve the overall quality of the generated face image, making the generated face image more realistic and clear. In addition, the loss function of the diffusion model is constructed based on facial feature characteristics, the diffusion model is trained in the reconstruction process based on the loss function, and the diffusion model is guided to pay more attention to the details of the facial feature characteristics, thereby improving the quality of the facial feature characteristics of the generated face image.

[0114] In some embodiments, a first clean image is obtained from a face data set as a training set, the first clean image in the training set is input into a diffusion process of a diffusion model, and Gaussian noise is gradually added in the diffusion process until the first clean image is completely covered by the Gaussian noise to obtain a noise image. An additional MSE loss is constructed based on prior information of facial feature, and a loss function for training of a diffusion network of the diffusion model is composed of a mean square error loss MSE and the additional MSE loss. The noise image is input into a reconstruction process of the diffusion model, the diffusion network is trained in the reconstruction process based on the loss function to obtain a target diffusion network, a random Gaussian noise image is obtained, and the random Gaussian noise image is input into the reconstruction process of the diffusion model to obtain a generated face image using the target diffusion network. Wherein, inputting the noise image into the reconstruction process of the diffusion model and training the diffusion network in the reconstruction process based on the loss function includes inputting the noise image into the reconstruction process of the diffusion model, using the diffusion network to gradually infer the added Gaussian noise in the noise image in the reconstruction process based on the loss function, and gradually removing the added Gaussian noise to obtain a second clean image. Wherein, constructing the additional MSE loss based on the prior information of facial feature includes: performing feature extraction on the first clean image to obtain facial feature and non-feature; performing inflation operation on the facial feature to cover all areas within a preset range around the facial feature, and performing inflation operation on the non-feature to cover the edge area; combining the facial feature and the non-feature after the inflation operation into a mask picture; performing blurring operation on the mask picture to obtain a target mask picture; and obtaining the additional MSE loss according to the target mask picture.

[0115] The diffusion network comprises a time step embedding vector generation module, an encoder, and a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, a Wavelet block, and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, the decoder third module comprises a ResBlock block, an Attention block, and an up-sampling layer connected in sequence, and the decoder fourth module comprises a ResBlock block, an up-sampling layer, and a ResBlock block connected in sequence, the Wavelet block is configured to extract a first frequency feature of a first feature map through a discrete wavelet transform module; the time step embedding vector generation module comprises a linear layer, a SiLU activation function, and a linear layer connected in sequence; the encoder comprises an encoder first module, an encoder second module, an encoder third module, and an encoder fourth module connected in sequence, the encoder first module comprises a ResBlock block and a down-sampling layer connected in sequence, the encoder second module comprises a ResBlock block and a down-sampling layer connected in sequence, the encoder third module comprises a ResBlock block, an Attention block, and a down-sampling layer connected in sequence, and the encoder fourth module comprises a ResBlock block, an Attention block, and a down-sampling layer connected in sequence. The Wavelet block comprises a discrete wavelet transform module, a ResBlock block, and an inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is configured to extract a first frequency feature of a first feature map, the ResBlock block is configured to separate Gaussian noise according to the frequency of the first frequency feature to obtain a second frequency feature, and the inverse discrete wavelet transform module is configured to convert the second frequency feature into a second feature map.

[0116] The embodiment of the present disclosure further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the image generation method based on the diffusion model when executing the computer program. The electronic device can be any intelligent terminal such as a computer.

[0117] The embodiment of the present disclosure further provides a storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image generation method based on the diffusion model.

[0118] The terms "first", "second", "third", "fourth" and the like in the description of the present disclosure and in the claims, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed as interchangeable in order to describe the embodiments of the present disclosure in relation to their functionality. Furthermore, the terms "comprising", "having", "including", and "containing" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises, has, includes or contains an item or list of items does not include only those items, but can include other items not expressly listed or inherent to such process, method, system, product, or apparatus.

[0119] It should be understood that, in the present disclosure, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0120] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. are not included in the number, above, below, within, etc. are understood to include the number.

[0121] It should also be understood that the various embodiments provided by the embodiments of the present disclosure can be combined in any manner to achieve different technical effects.

[0122] The above is a specific description of the preferred embodiments of the present disclosure, but the present disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present disclosure, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for generating an image based on a diffusion model, characterized by, The method comprises: obtaining a first clean image in a face data set; inputting the first clean image into a diffusion process of a diffusion model, and gradually adding Gaussian noise in the diffusion process to obtain a noisy image; the diffusion model comprises a time step embedding vector generation module, an encoder, and a decoder, the decoder comprises a decoder first module, a decoder second module, a decoder third module, and a decoder fourth module connected in sequence, the decoder first module comprises a Wavelet block, an Attention block, the Wavelet block, and an up-sampling layer connected in sequence, the decoder second module comprises a ResBlock block, the Attention block, and the up-sampling layer connected in sequence, the decoder third module comprises the ResBlock block, the Attention block, and the up-sampling layer connected in sequence, and the decoder fourth module comprises the ResBlock block, the up-sampling layer, and the ResBlock block connected in sequence, and the Wavelet block is configured to extract first frequency features of a first feature map through a discrete wavelet transform module; constructing a loss function of the diffusion model based on facial feature characteristics; inputting the noisy image into a reconstruction process of the diffusion model, training the diffusion model in the reconstruction process based on the loss function, and obtaining a target diffusion model; obtaining a random Gaussian noise image, inputting the random Gaussian noise image into the reconstruction process of the target diffusion model, and obtaining a generated face image; the method of inputting the first clean image into a diffusion process of a diffusion model, and gradually adding Gaussian noise in the diffusion process to obtain a noisy image comprises: the method of inputting the first clean image into the diffusion process of the diffusion model, and gradually adding the Gaussian noise in the diffusion process by a diffusion formula to obtain the noisy image, the diffusion formula being: ， wherein , , is the first clean image, is the noisy image after t steps of adding the Gaussian noise, is the Gaussian noise, t .

2. The image generation method based on a diffusion model according to claim 1, wherein, the method of inputting the noisy image into a reconstruction process of the diffusion model, and training the diffusion model in the reconstruction process based on the loss function comprises: the method of inputting the noisy image into the reconstruction process of the diffusion model, and gradually inferring the added Gaussian noise in the noisy image by a reconstruction formula in the reconstruction process based on the loss function, and gradually removing the added Gaussian noise to obtain a second clean image, the reconstruction formula being: ， wherein is the noisy image after adding the Gaussian noise by t steps, is the noisy image after adding the Gaussian noise by t steps, is the noisy image after adding the Gaussian noise by t steps, , , t , is the added Gaussian noise estimated under the parameters .

3. The method of claim 1, wherein, the method of constructing a loss function of the diffusion model based on facial feature characteristics comprises: constructing an additional MSE loss based on the facial feature characteristics; combining a mean square error loss MSE and the additional MSE loss to form the loss function of the diffusion model, the loss function of the diffusion model being: ， wherein is the loss function for the diffusion model, is the mean squared error loss MSE, is the additional MSE loss.

4. The image generation method based on a diffusion model according to claim 3, characterized in that, the method of constructing an additional MSE loss based on the facial feature characteristics comprises: performing feature extraction on the first clean image to obtain the facial feature characteristics and non-feature characteristics; performing an inflation operation on the facial feature characteristics to cover all regions within a preset range around the facial feature characteristics, and performing the inflation operation on the non-feature characteristics to cover edge regions; The face feature and the non-face feature after the expansion operation are combined into a mask picture; The mask picture is blurred to obtain a target mask picture; The target mask picture is used to obtain the additional MSE loss: ， wherein, is the first clean image, is the noisy image after adding the Gaussian noise for t steps, is the Gaussian noise, is the added Gaussian noise estimated under the parameters, is the added Gaussian noise estimated under the parameters, is the target mask picture, t .

5. The method of claim 3, wherein, The mean square error loss MSE is: ， wherein is the first clean image, is the noisy image after adding the Gaussian noise for t steps, is the Gaussian noise, is the estimated added Gaussian noise under the parameters is the estimated added Gaussian noise under the parameters 6. The method of claim 1, wherein, The Wavelet block comprises the discrete wavelet transform module, the ResBlock block and the inverse discrete wavelet transform module connected in sequence; the discrete wavelet transform module is used to extract the first frequency feature of the first feature map; the ResBlock block is used to separate the Gaussian noise according to the frequency of the first frequency feature to obtain a second frequency feature; and the inverse discrete wavelet transform module is used to convert the second frequency feature into a second feature map.

7. The method of claim 1, wherein, The time step embedding vector generation module comprises linear layers, SiLU activation functions and linear layers connected in sequence; the encoder comprises an encoder first module, an encoder second module, an encoder third module and an encoder fourth module connected in sequence; the encoder first module comprises the ResBlock block and a down-sampling layer connected in sequence; the encoder second module comprises the ResBlock block and the down-sampling layer connected in sequence; the encoder third module comprises the ResBlock block, the Attention block and the down-sampling layer connected in sequence; and the encoder fourth module comprises the ResBlock block, the Attention block and the down-sampling layer connected in sequence.

8. An electronic device, comprising: The electronic device comprises a memory and a processor; the memory stores a computer program; and the processor executes the computer program to implement the image generation method based on the diffusion model according to any one of claims 1 to 7.

9. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the image generation method based on the diffusion model according to any one of claims 1 to 7.

Images