Generative de-noising device training and controllable generation method based on diffusion model distillation

Abstract

The invention provides a training method of a generative denoising device and an image controllable generation and restoration method. The training method comprises the following steps: taking a pre-trained diffusion model as a teacher diffusion model; initializing a generative denoising device, a score model and a discriminator; obtaining a noiseless signal, and obtaining a noisy signal and Gaussian noise in combination with a generative denoising device; obtaining effective noise intensity and effective noise signals according to the noisy signals and the Gaussian noise; estimating a data score by using a teacher diffusion model according to the effective noise intensity, the effective noise signal and the noisy signal; according to the noisy signal, estimating a model score by using a score model; calculating and optimizing the gradient of learnable parameters of the generative denoising device; and according to the noisy signal, the data score, the model score and the optimized generative de-noising device, calculating de-noising score matching of the score model and adversarial loss of the discriminator, and optimizing the score model and the discriminator. And realizing image controllable generation and image restoration based on the generative de-noising device.

Description

Technical Field

[0001] The present application relates to the field of signal processing, and in particular, to a generative denoiser training method based on diffusion model distillation, an image controllable generation method, an image processing system, a terminal and a medium. Background Art

[0002] With the development of deep learning technology, generative models have achieved remarkable results in fields such as image and video generation. Diffusion models, as an emerging generative model, have attracted widespread attention due to their outstanding performance in generating high-dimensional data. Diffusion models model complex data distributions by gradually adding noise to the data and then learning how to reverse this process to generate data. However, while the iterative sampling mechanism of diffusion models provides flexible inference control, its low sampling efficiency in practical applications, especially in scenarios requiring real-time generation, limits its widespread use.

[0003] To address this issue, researchers have proposed various methods to accelerate the sampling process of diffusion models. One approach involves distilling a pre-trained diffusion model into a faster generator through knowledge distillation. However, existing distillation methods typically only support sampling for one or a few steps, which compromises the quality and diversity of generated samples. Furthermore, these methods often lack fine-grained control over the generation process when handling complex generation tasks, resulting in insufficient controllability and flexibility in the generated results. Summary of the Invention

[0004] In response to one of the defects in the prior art, the purpose of this application is to provide a generative denoiser training method, image generation and restoration method based on diffusion model distillation.

[0005] In a first aspect of the present application, a method for training a generative denoiser based on diffusion model distillation is provided, comprising:

[0006] Use the pre-trained diffusion model as the teacher diffusion model;

[0007] Initialize the generative denoiser, scoring model and discriminator;

[0008] Acquire a noise-free signal and combine it with the generative denoiser to obtain a noisy signal and Gaussian noise;

[0009] Obtaining an effective noise intensity and an effective noise signal according to the noisy signal and the Gaussian noise;

[0010] estimating a data score using the teacher diffusion model according to the effective noise intensity, the effective noise signal, and the noisy signal;

[0011] estimating a model score using the scoring model according to the noisy signal;

[0012] Calculate and optimize the gradient of the learnable parameters of the generative denoiser based on the noisy signal, the data score, the model score and the discriminator;

[0013] Calculating the denoising score match of the scoring model and the adversarial loss of the discriminator based on the noisy signal, the data score, the model score, and the optimized generative denoiser, and optimizing the scoring model and the discriminator;

[0014] Repeat the above steps until convergence.

[0015] Optionally, the generative denoiser obtained through training is μ θ (x0|y σ ), y σ Represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; using neural network Parameterized μ θ (x0|y σ ), represents the intermediate noisy signal, represents the intermediate noise intensity; θ is D θ The learnable parameters are used to initialize the neural network with the teacher diffusion model D0 Output of the neural network Satisfy x0~μ θ (x0|y σ );

[0016] The generative denoiser generates a noise-free signal based on a noisy signal, comprising:

[0017] Get the first noisy signal y σ and the intensity σ of the first Gaussian noise;

[0018] The noise intensity is amplified by the scaling factor γ to obtain the intermediate noise intensity.

[0019] Get the standard Gaussian noise z by sampling from a standard Gaussian distribution with an expectation of 0 and a variance of 1, and scale z And add the first noisy signal y σ , get the intermediate noisy signal

[0020] The first noisy signal y σ And the intensity σ of the first Gaussian noise is input into the trainable ControlNet, and the trainable ControlNet encodes the first noisy signal y through the encoder networkσ with the first Gaussian noise intensity σ to preserve the intermediate noisy signal Key control information lost due to noise;

[0021] According to the intermediate noisy signal And the key control information, through the teacher diffusion model D with parameter θ θ , get the denoised signal x θ .

[0022] Optionally, the scoring model is D φ (x t ,t,y σ ,σ),y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input, x t represents the second noise signal, t represents the noise intensity t of the second Gaussian noise, and φ is D φ The learnable parameters are initialized with the teacher diffusion model D0;

[0023] The score model is used to estimate the score function of the generative denoiser, including:

[0024] The first noise signal y σ and the noise intensity σ of the first Gaussian noise is input to the conditional encoder;

[0025] The second noise signal x t and the noise intensity t of the second Gaussian noise is input to the backbone encoder;

[0026] Combine the output of the backbone encoder and the output of the conditional encoder using a learnable amplitude-preserving addition layer;

[0027] The output of the amplitude-preserving addition layer is input to a decoder to obtain an estimate of the score function of the generative denoiser.

[0028] Optionally, the discriminator is C ψ (x t ,t,y σ ,σ),y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; x t represents the second noise signal, t represents the noise intensity t of the second Gaussian noise; ψ = [ψ1, ψ2] is the learnable parameter of the discriminator;

[0029] The discriminator uses a generative sample encoder and real sample encoder From the generated samples (x t,t) and true samples (y σ ,σ) to extract features and distinguish them, including:

[0030] Generate sample using encoder Extract the generated samples of the generative denoiser (x t ,t) features;

[0031] Using real sample encoders Extract the real sample (y σ ,σ) features;

[0032] Generate sample features and the real sample characteristics Splicing in the channel dimension to obtain splicing features;

[0033] The concatenated features are sequentially passed through an average pooling layer and a linear layer, and a Sigmoid nonlinear activation function is used to output a classification probability within [0, 1].

[0034] Optionally, the acquiring of the noise-free signal and combining the generative denoiser to obtain the noisy signal and Gaussian noise includes:

[0035] Get the noise-free signal x0;

[0036] The first Gaussian noise with a noise intensity of σ is applied to the noise-free signal x0, starting from a Gaussian noise with a mean of x0 and a variance of σ 2 The first noisy signal y is obtained by sampling the Gaussian distribution σ , where σ is sampled from the predefined first Gaussian noise intensity distribution;

[0037] The first noisy signal y σ Through the generative denoiser μ θ (x0|y σ ), and obtain the denoised signal x θ ;

[0038] For the denoised signal x θ Apply a second Gaussian noise of noise intensity t, starting from the mean x θ , the variance is t 2 The second noisy signal x is obtained by Gaussian distribution sampling t , where t is sampled from a predefined second Gaussian noise intensity distribution.

[0039] Optionally, obtaining an effective noise intensity and an effective noise signal according to the noisy signal and the Gaussian noise includes:

[0040] The effective noise intensity σ is calculated based on the noise intensity σ of the first Gaussian noise and the noise intensity t of the second Gaussian noise.eff , the effective noise intensity σ eff The square inverse of is the sum of the square inverse of the noise intensity σ of the first Gaussian noise and the square inverse of the noise intensity t of the second Gaussian noise;

[0041] The effective noise intensity σ eff The square of the ratio of the noise intensity σ of the first Gaussian noise to the noise intensity t of the second Gaussian noise is the weighted combination of the first noisy signal y σ and the second noisy signal x t , to obtain the effective noise signal

[0042] Optionally, estimating a data score using the teacher diffusion model according to the effective noise intensity, the effective noise signal, and the noisy signal includes:

[0043] The output of the teacher diffusion model is obtained according to the effective noise intensity and the effective noise signal, and the ... t Subtract and scale with the square of the noise intensity t of the second Gaussian noise to estimate the data score s0.

[0044] Optionally, estimating a model score using the scoring model according to the noisy signal includes:

[0045] According to the first noisy signal y σ and the noise intensity σ of the first Gaussian noise, the second noisy signal x t and the noise intensity t of the second Gaussian noise to obtain the output of the score model, and the second noisy signal x t Subtract and scale by the square of the noise intensity t of the second Gaussian noise to estimate the model score s φ .

[0046] Optionally, calculating and optimizing the gradient of the learnable parameters of the generative denoiser based on the noisy signal, the data score, the model score and the discriminator includes:

[0047] Calculate the loss function of the generative denoiser The gradient g with respect to the learnable parameters θ of the generative denoiser NCVSD ;

[0048] Calculate the logarithmic gradient g of the discriminator with respect to the learnable parameters θ of the generative denoiser C ;

[0049] According to the gradient g of the generative denoiser NCVSD and the gradient g of the discriminator C , gradient descent is used to optimize the learnable parameters θ of the generative denoiser.

[0050] Optionally, calculating the denoising score match of the scoring model and the adversarial loss of the discriminator based on the noisy signal, the data score, the model score, and the optimized generative denoiser, and optimizing the scoring model and the discriminator includes:

[0051] The scoring model is trained synchronously with the model score matching during the generative denoiser optimization process:

[0052] The discriminator is optimized using cross entropy loss during the optimization of the generative denoiser.

[0053] A second aspect of the present application provides a method for controllable image generation, comprising:

[0054] Based on pre-trained generative denoiser μ θ (x0|y σ ), given the energy function ε(x0,c) used to measure the degree of matching between a single image sample x0 and a given condition c;

[0055] Distribution that is never normalized Sampling in the middle, obtaining the sample x0 that satisfies the given condition c, that is, realizing the controllable generation of the image, where q data (x0) is the data distribution.

[0056] Optionally, the unnormalized distribution Medium sampling, including:

[0057] Gradually reduce the noise intensity σ, and alternately use the generative denoiser to approximate the noise intensity from π σ (x0|u) is sampled and Langevin's algorithm is used to extract the value from π σ Sampling (u|x0) until the noise intensity σ is less than a given threshold;

[0058] Among them, the generative denoiser is used to approximate the σ (x0|u) sampling:

[0059]

[0060] Among them, using Lang Zhiwan algorithm from π σ Sampling in (x|x0):

[0061]

[0062] π σ (x0|u) represents a distribution, u is the noise signal, x0 is the noise-free signal, that is, the noise-free signal x0 is generated from the noise signal u; π σ(u|x0) means that the noise signal u is obtained from the noise-free signal x0, and c is a given condition;

[0063] Reduce the σ value;

[0064] Repeat the above steps until the σ value is less than the given threshold, and output the x0 of the last step as the generated image.

[0065] Optionally, the generative denoiser is used to approximate from π σ (x0|u) sampling, including:

[0066] Calculate the effective noise intensity:

[0067]

[0068] Calculate the effective noise signal:

[0069]

[0070] from Sample x0;

[0071] From q(x i-1 ∣x i ,x0) sample x i-1 ;

[0072] Repeat the above steps until i=2; where x i The latent variable introduced for multi-step sampling, σ i is the noise intensity of the latent variable, y σ Represents a noisy signal

[0073] The Langevin algorithm is used to σ Sampling in (u|x0) includes:

[0074]

[0075] Among them, γ σ =C1(β -1 C2+σ -2 ) -1 is the adaptive step size; C1, C2, β are adjustable hyper parameters; u k To realize the intermediate iterative process of sampling u based on Langevin algorithm, if the total number of iterations is K, use u K as the sampling sample of u; σ is π σ The noise intensity of (u|x0).

[0076] Optionally, the given condition c includes one or more of the following features:

[0077] The text description related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the text description c and the content of image x0;

[0078] The speech related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the speech c and the content of image x0;

[0079] Semantic information that matches the content of image x0. In this case, the energy function ε(x0,c) is defined as the degree of correlation between the semantic information c and the content of image x0.

[0080] Degraded image y associated with image x0;

[0081] In the image deblurring scenario, the degraded image y is the image x0 blurred by the blur kernel;

[0082] In the image super-resolution scenario, the degraded image y is the image after downsampling of image x0;

[0083] In the image restoration scenario, the degraded image y is the image obtained by setting some pixel values of the image x0 to zero;

[0084] In the Fourier phase retrieval scenario, the degraded image y is the image obtained by zero-padding, Fourier transform, and modulo operation of image x0;

[0085] At this time, the energy function ε(x0,y) is defined as A represents the corresponding degradation operation in the above scenario.

[0086] A third aspect of the present application provides an image processing system, comprising:

[0087] Data module, obtains the image to be processed;

[0088] The processing module is a plug-and-play generative denoiser obtained by the method described above, which processes the image to be processed to achieve controllable generation of the image.

[0089] The fourth aspect of the present application provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the terminal can be used to execute the generative denoiser training method based on diffusion model distillation, or the controllable image generation method; and can be used to operate the system.

[0090] In a fifth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, it can be used to execute the generative denoiser training method based on diffusion model distillation, or the controllable image generation method; it can be used to run the system.

[0091] The generative denoiser training method based on diffusion model distillation provided in this application adopts the technical means of revealing the intrinsic connection between the scoring model and the denoising posterior distribution scoring model to estimate the model score, and integrating this intrinsic connection into the distillation framework to train the generator, resulting in the technical effect that the trained generator can quickly generate data while retaining the iterative sampling capability of the diffusion model.

[0092] Other technical effects brought about by the additional features will be further explained in the corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0094] Figure 1 4 is a flowchart of a method for training a generative denoiser based on diffusion model distillation according to an exemplary embodiment;

[0095] Figure 2 is a framework diagram of an image processing system according to an exemplary embodiment;

[0096] Figure 3 A flowchart of image generation according to an exemplary embodiment is shown;

[0097] Figure 4 The figure is a flowchart of image restoration according to an exemplary embodiment. DETAILED DESCRIPTION

[0098] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention. These modifications and improvements fall within the scope of protection of the present invention. Parts not described in detail below may be implemented using existing technologies.

[0099] Explanation of terms:

[0100] The generative denoiser is mathematically defined as a distribution that fits the posterior distribution of the original clean image given a noisy image. Its input is the noisy image and the noise intensity of the image, and its output is an estimate of the original clean image. For the same input, the output of the model will be different each time, which is used to simulate the randomness of sampling in the posterior distribution. In the embodiment of the application, the parameterized distribution μ of the generative denoiser is θ (x0|y σ ) can use neural network To express.

[0101] In the prior art, the diffusion model generation speed is slow, and the acceleration method does not support controllable generation. Based on the above problems, the embodiment of the present application provides a generative denoiser training method based on diffusion model distillation to solve the above problems.

[0102] like Figure 1 As shown, in one embodiment of the present application, a generative denoiser training method based on diffusion model distillation includes the following steps:

[0103] S100, the pre-trained diffusion model is used as the teacher diffusion model;

[0104] S200, initialize the generative denoiser, scoring model and discriminator;

[0105] S300, obtain the noise-free signal and combine it with the generative denoiser to obtain the noisy signal and Gaussian noise;

[0106] S400, obtaining an effective noise intensity and an effective noise signal according to the noisy signal and the Gaussian noise;

[0107] S500, estimating the data score using the teacher diffusion model according to the effective noise intensity, the effective noise signal and the noisy signal;

[0108] S600, estimating a model score using a scoring model according to the noisy signal;

[0109] S700, calculating and optimizing the gradient of the learnable parameters of the generative denoiser based on the noisy signal, the data score, the model score and the discriminator;

[0110] S800 calculates the denoising score match of the scoring model and the adversarial loss of the discriminator based on the noisy signal, data score, model score, and the optimized generative denoiser, and optimizes the scoring model and discriminator.

[0111] Repeat the above steps S300-S800 until convergence to obtain a trained generative denoiser.

[0112] Specifically, the generative denoiser is the student model. Through the above process, the relatively simple student model can learn the pattern of the relatively complex teacher diffusion model.

[0113] The above examples of this application reveal the intrinsic connection between the scoring model and the denoised posterior distribution score model to estimate the model score, and integrate this intrinsic connection into the distillation framework to train the generator. The resulting trained generator can not only generate high-quality samples in a single sampling step, but also further improve sample quality through multi-step sampling. It also supports plug-and-play, providing flexible and controllable sampling capabilities for various downstream tasks.

[0114] In order to better obtain a noise-free signal, in some specific embodiments of the present application, the generative denoiser can be parameterized by a neural network as y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; represents the intermediate noisy signal, represents the intermediate noise intensity; θ is μ θ The learnable parameters are initialized with the teacher model D0.

[0115] The generative denoiser generates a noise-free signal based on a noisy signal. The specific operation is:

[0116] S211, obtaining a first noisy signal y σ and the intensity σ of the first Gaussian noise;

[0117] S212, amplify the noise intensity using the scaling factor γ to obtain an intermediate noise intensity

[0118] S213, obtain standard Gaussian noise z by sampling from a standard Gaussian distribution with an expectation of 0 and a variance of 1, and scale z And add the first noisy signal y σ , get the intermediate noisy signal The specific formula is:

[0119]

[0120] S214, the first noisy signal y σ The intensity σ of the first Gaussian noise is input into the trainable ControlNet to extract key control information;

[0121] S215, according to the middle noisy signal and key control information, through the teacher model D with parameter θ θ Get the denoised signal x θ .

[0122] The above embodiments of the present application provide a scalable training method for a generative denoiser, which can be applied to large-scale, high-resolution image datasets.

[0123] In order to calculate the gradient data in the training phase, a scoring model is introduced. In some specific embodiments of the present application, the scoring model is D φ (x t ,t,y σ ,σ),u σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input, x trepresents the second noise signal, t represents the noise intensity t of the second Gaussian noise, and φ is D φ The learnable parameters are initialized with the teacher diffusion model D0.

[0124] The scoring model is used to estimate the score function of the generative denoiser The specific operations are:

[0125] S221, the first noise signal y σ The noise intensity σ of the first Gaussian noise is input into the conditional encoder, and the conditional encoder is the encoder of the U-Net model;

[0126] S222, the second noise signal x t The noise intensity t of the second Gaussian noise is input into the backbone encoder, which is the encoder of the U-Net model;

[0127] S223, using a learnable amplitude-preserving addition layer MPSum w (a, b) combines the output a of the backbone encoder and the output b of the conditional encoder. The specific formula is:

[0128]

[0129] Where: w∈[0,1] is the learnable weight, initialized to 0 to prevent destroying the knowledge of the pre-trained model;

[0130] S224, input the output of the amplitude-preserving addition layer into the decoder to obtain the score function of the generative denoiser The decoder is the decoder of the U-Net model.

[0131] The above-mentioned embodiments of the present application effectively parameterize the scoring model and can efficiently estimate the model parameters for implementing the training of the generative denoiser.

[0132] In order to enhance the authenticity and quality of the generated samples, the discriminator in the generative adversarial training is introduced as an auxiliary model during training. In some specific embodiments of the present application, the discriminator is C ψ (x t ,t,y σ ,σ),y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; represents the intermediate noisy signal, represents the intermediate noise intensity; ψ=[ψ1,ψ2] is the learnable parameter of the discriminator.

[0133] The discriminator uses the generated sample encoder and real sample encoder Generate samples from the generative denoiser (x t ,t) and true samples (y σ ,σ) to extract features and distinguish them. The specific operations are as follows:

[0134] S231, using the generated sample encoder Extract the generated samples of the generative denoiser (x t ,t) features, the generated sample encoder is the encoder of the U-Net model, which is initialized by the encoder of the teacher diffusion model D0;

[0135] Using real sample encoders Extract the real sample (y σ ,σ) features, the real sample encoder is the encoder of the U-Net model, which is initialized by the encoder of the teacher diffusion model D0;

[0136] S232, will generate sample features and real sample characteristics Splicing in the channel dimension to obtain splicing features;

[0137] S233, the concatenated features are sequentially passed through an average pooling layer and a linear layer, and a classification probability within [0, 1] is output through the Sigmoid nonlinear activation function.

[0138] The above embodiments of the present application effectively parameterize the discriminator to implement generative adversarial training of the generative denoiser to further improve sample quality.

[0139] In some specific embodiments of the present application, in S300, obtaining a noise-free signal, applying a first Gaussian noise to obtain a first noisy signal, denoising the noisy signal using a generative denoiser to obtain a denoised signal, and applying a second Gaussian noise to the denoised signal to obtain a second noisy signal may be performed by the following steps:

[0140] S301, obtaining a noise-free signal x0;

[0141] S302, applying the first Gaussian noise of noise intensity σ to the noise-free signal x0, starting from the mean x0 and variance σ 2 The first noisy signal y is obtained by sampling the Gaussian distribution σ , where σ is sampled from the predefined first Gaussian noise intensity distribution;

[0142] S303: The first noisy signal y σ Through the generative denoiser μ θ (x0|y σ ), and obtain the denoised signal x θ ;

[0143] For the denoised signal x θ Apply a second Gaussian noise of noise intensity t, starting from the mean x θ , the variance is t 2 The second noisy signal x is obtained by Gaussian distribution sampling t , where t is sampled from a predefined second Gaussian noise intensity distribution.

[0144] The two sets of noisy signals and Gaussian noise obtained in the above embodiments of the present application can be used for subsequent calculation of effective noise signals.

[0145] In some specific implementations of the present application, in S400, the effective noise signal is obtained by calculating according to the first noisy signal, the first Gaussian noise, the second noisy signal, and the second Gaussian noise, and the following steps may be used:

[0146] S401, calculating an effective noise intensity σ according to the noise intensity σ of the first Gaussian noise and the noise intensity t of the second Gaussian noise. eff , effective noise intensity σ eff The square inverse of is the sum of the square inverse of the noise intensity σ of the first Gaussian noise and the square inverse of the noise intensity t of the second Gaussian noise. The specific formula is:

[0147] σ eff =(σ -2 +t -2 ) -1 / 2

[0148] S402, the effective noise intensity σ eff The square of the ratio of the noise intensity σ of the first Gaussian noise to the noise intensity t of the second Gaussian noise is the weighted combination of the first noisy signal y σ and the second noisy signal x t , to obtain the effective noise signal The specific formula is:

[0149]

[0150] Among them, x i The latent variable introduced for multi-step sampling, σ i is the noise intensity of the latent variable, y σ Represents a noisy signal.

[0151] In the above embodiments of the present application, an effective noise signal and an effective noise intensity are obtained for estimating a data score.

[0152] In some specific implementations of the present application, in S500, estimating the data score using the teacher diffusion model based on the effective noise signal and the second noisy signal may be performed by the following steps:

[0153] According to the effective noise intensity σeff and the effective noise signal Get the output of the teacher diffusion model D0 and the second noisy signal x t Subtract and scale by the square of the noise intensity t of the second Gaussian noise to estimate the data score s0. The specific formula is:

[0154]

[0155] In the above-mentioned embodiments of the present application, the data score is effectively estimated based on the pre-trained diffusion model, providing a data basis for the subsequent calculation of the gradient.

[0156] In some specific embodiments of the present application, in S600, estimating a model score using a scoring model based on the first noisy signal and the second noisy signal may include the following steps:

[0157] According to the first noisy signal y σ And the noise intensity σ of the first Gaussian noise, the second noisy signal x t And the noise intensity t of the second Gaussian noise is used to obtain the score model D φ The output of the second noisy signal x t Subtract and scale by the square of the noise intensity t of the second Gaussian noise to estimate the model score s φ , the specific formula is:

[0158] s φ =t -2 (D φ (x t ,t,y σ ,σ)-x t )

[0159] The above-mentioned embodiment of the present application effectively estimates the model score based on the scoring model, providing a data basis for the subsequent calculation of the gradient.

[0160] In some specific embodiments of the present application, in S700, the gradient of the learnable parameters of the generative denoiser is calculated based on the first noisy signal, the second noisy signal, the data score, the model score, and the discriminator, and the generative denoiser is optimized based on gradient descent. The following steps may be employed:

[0161] S701, calculate the loss function of the generative denoiser Where q is the posterior distribution of the original clean image after the real given noisy image, D KL is the KL (Kullback-Leibler) divergence, μ θ μ θ (x0|y σ ). The gradient g of the learnable parameters θ of the generative denoiser NCVSD, a neural network with a parameterized generative denoiser Gradients with respect to learnable parameters θ and model scores s φ It is obtained by multiplying the difference with the data score s0. The specific formula is:

[0162]

[0163] S702, calculate the logarithmic gradient g of the discriminator with respect to the learnable parameter θ of the generative denoiser C , the specific formula is:

[0164]

[0165] S703, according to the gradient g of the generative denoiser NCVSD and the gradient g of the discriminator C , we use gradient descent to optimize the learnable parameters θ of the generative denoiser:

[0166] θ←θ-α ncvsd ·g NCVSD +α gan ·g C

[0167] Where: α ncvsd and α gan They are respectively used to adjust the learning rate of the corresponding gradient, which is determined by the preset strategy.

[0168] In the above embodiment of the present application, the logarithmic gradient of the learnable parameters of the generative denoiser is calculated to update the parameters of the generative denoiser.

[0169] In some specific embodiments of the present application, in S800, based on the first noisy signal, the second noisy signal, the data score, the model score, and the optimized generative denoiser, the denoising score match of the scoring model and the adversarial loss of the discriminator are calculated, and the scoring model and the discriminator are optimized based on gradient descent. The following steps may be employed:

[0170] S801, scoring model D φ In the generative denoiser μ θ During the optimization process, denoising score matching training is used simultaneously:

[0171]

[0172] in, represents the expectation, μ θ (x0|y σ ) represents the generative denoiser, The mean is x0 and the variance is t 2 Gaussian distribution with parameter φ for the score model D φ The input is the first noisy signal yσ And the noise intensity σ of the first Gaussian noise, the second noisy signal x t and the noise intensity t of the second Gaussian noise.

[0173] S802, Discriminator C ψ In the generative denoiser μ θ During the optimization process, cross entropy loss optimization is used:

[0174]

[0175] in, Refers to the expectation of data distribution, where q refers to μ θ (x0|y σ ); Refers to the expectation of the generative denoiser, where μ θ μ θ (x0|y σ ); log represents the logarithmic sign; ψ represents the discriminator C ψ Parameters.

[0176] The above-mentioned embodiments of the present application optimize the discriminator parameters so that it can correctly distinguish between samples generated by the generator and real data samples.

[0177] Based on the same technical concept, in some specific embodiments of the present application, a method for controllable image generation, such as Figure 2 As shown, the following steps are included:

[0178] Based on pre-trained generative denoiser μ θ (x0|y σ ), given the energy function ε(x0,c) used to measure the degree of matching between a single image sample x0 and a given condition c;

[0179] Distribution that is never normalized By sampling, we can obtain samples x0 that meet the given condition c, that is, we can realize the controllable generation of the image, where q data (x0) is the data distribution.

[0180] Specifically, "controllable" means that the results are controlled by given conditions, that is, they satisfy given conditions, including image classification, image description, damaged image restoration, etc.

[0181] This application utilizes a trained generative denoiser for image generation. Compared to diffusion-based image generation techniques, this significantly reduces the number of samples required, significantly improving data generation efficiency and conserving computing resources. This generative denoiser enables faster and more efficient image processing and restoration, accelerating sampling and improving sample quality.

[0182] In order to achieve better controllability in some specific embodiments of the present application, the distribution of the unnormalized The sampling is implemented as follows:

[0183] Gradually reduce σ and alternately use the generative denoiser to approximate the value from π under a given σ. σ (x0|u) is sampled and Langevin's algorithm is used to extract the σ Sample (u|x0) until it is less than a given threshold. The following steps can be used:

[0184] S901, using a generative denoiser to approximate the σ (x0|u) sampling,

[0185]

[0186] S902, using Langevin algorithm from π σ Sampling in (u|x0),

[0187]

[0188] S903, reduce the σ value;

[0189] Repeat the above steps until the σ value is less than the given threshold, and output x0 as the generated image.

[0190] Specifically, S901 uses a generative denoiser to approximate the σ (x0|u) sampling, including:

[0191] S9011, calculate the effective noise intensity:

[0192]

[0193] S9012, calculate the effective noise signal:

[0194]

[0195] S9013, from Sample x0;

[0196] S9014, from q(x i-1 ∣x i ,x0) sample x i-1 ;

[0197] Repeat the above steps until i=2.

[0198] Specifically, in S902, Langevin algorithm is used to calculate the σ Sampling in (u|x0), using adaptive step size to implement Langevin dynamics based on Euler method discretization:

[0199]

[0200] Among them, γ σ =C1(β -1 C2+σ -2 ) -1 This is the adaptive step size used in the embodiment of the present application.

[0201] The above implementation of the present application effectively realizes the σ (u|x0) sampling and adaptive step size ensure the convergence of the Langevin algorithm. The Langevin algorithm includes constraints on the condition and image matching degree ε(u, c), which can ensure that the output satisfies the given condition c and achieve controllable image generation.

[0202] In order to accomplish more tasks, in some specific implementations of this application, the given condition c may include the following situations:

[0203] The text description related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the text description c and the content of image x0;

[0204] The speech related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the speech c and the content of image x0;

[0205] Semantic information that matches the content of image x0. In this case, the energy function ε(x0,c) is defined as the degree of correlation between the semantic information c and the content of image x0.

[0206] The degraded image y associated with image x0:

[0207] In the image deblurring scenario, the degraded image y is the image x0 blurred by the blur kernel;

[0208] In the image super-resolution scenario, the degraded image y is the image after downsampling of image x0;

[0209] In the image restoration scenario, the degraded image y is the image obtained by setting some pixel values of the image x0 to zero;

[0210] In the Fourier phase retrieval scenario, the degraded image y is the image obtained by zero-padding, Fourier transform, and modulo operation of image x0;

[0211] At this time, the energy function ε(x0,y) is defined as a represents the corresponding degradation operation in the above scenario.

[0212] Exemplary, an image restoration method based on a pre-trained generative denoiser μ θ (x0|yσ ), used in the unnormalized distribution Sampling in the middle, where q data (x0) is the data distribution, and ε(x0,y) is defined as Where y is the degraded image and A is the degradation operator.

[0213] Distribution that is never normalized The sampling is achieved by gradually reducing σ and using the generative denoiser to approximate the value from π under a given σ. σ (x0|u) is sampled and Langevin's algorithm is used to extract the σ Sample (u|x0) until it is less than a given threshold. The following steps can be used:

[0214] S1001, using a generative denoiser to approximate the σ (x0|u) sampling,

[0215]

[0216] S1002, using Langevin's algorithm from π σ Sampling in (u|x0),

[0217]

[0218] S1003, reduce the σ value;

[0219] Repeat the above steps until the σ value is less than the given threshold, and output x0 as the generated image.

[0220] Based on the same technical concept, an image processing system such as Figure 2 As shown, including:

[0221] Data module, obtains the image to be processed;

[0222] The processing module plugs and plays the generative denoiser obtained by any of the methods to process the image to be processed and realizes controllable generation of the image.

[0223] The trained generative denoiser of the present application can achieve plug-and-play effects for various scenarios and has good flexibility without the need for additional training and additional data collection and processing, thus saving computing resources and human resources.

[0224] The preferred features of the above embodiments can be used alone in any embodiment, or in any combination without conflict. In addition, parts not described in detail in the embodiments can be implemented using existing technologies.

[0225] The following further illustrates the present application in conjunction with specific application examples / comparative examples to facilitate a better understanding of the above technical solutions of the present application. It should be understood that the following are merely partial examples and are not intended to limit the present application.

[0226] Comparative Example 1:

[0227] In this embodiment, a trained generative denoiser is used to generate natural images, such as Figure 3 As shown, the following steps are included:

[0228] S101, initialization: the current iteration step i is initialized to the initial step N, sampling from the Gaussian distribution of the initial noise intensity to obtain the auxiliary variable

[0229] S102, Generative Denoising Module: Use the generative denoiser to remove the noise of the auxiliary variable u and update the clean sample estimate get

[0230] S103, annealing, i decreases by 1, and the current noise intensity is set to σ i , if the current iteration step i = 1, then the clean sample estimate is output Otherwise, the clean sample is estimated Strengthening σ i Gaussian noise is obtained to obtain u, and returned to S102.

[0231] In this comparative example, natural image generation is achieved, where natural images are collected from the ImageNet dataset. Three other image generation methods based on diffusion and consistency models are compared: EDM2, proposed in "Analyzing and Improving the Training Dynamics of Diffusion Models," published at the 2024 Conference on Computer Vision and Pattern Recognition (CVPR); ECM, proposed in "Consistency Models Made Easy," published at the 2024 International Conference on on Learning Representations (ICLR); and sCM, proposed in "Simplifying, Stabilizing, and Scaling Continuous-time Consistency Models," published at the 2024 International Conference on on Learning Representations (ICLR). Image generation quality is evaluated using the FID (lower is better) and the number of neural network calls (NFE, lower is better).

[0232] The experimental results show that under the experimental conditions of a resolution of 64x64, the FID of the natural image generated by the above-mentioned image generation method of the present application using 4NFE at a model scale of L (model scale from small to large is L, XL) is 1.53, which is smaller than the FID of ECM-XL (1.96) and the FID of sCM-XL (1.66), and is close to the FID (1.33) of the 63NFE teacher diffusion model EDM2-XL; under the experimental conditions of a resolution of 512x512, the FID of the natural image generated by the above-mentioned image generation method of the present application using 4NFE at a model scale of L (model scale from small to large is L, XL, XXL) is 1.76, which is smaller than the FID (1.88) of sCM-XXL and the FID (1.81) of the 126NFE teacher diffusion model EDM2-XXL.

[0233] Comparative Example 2:

[0234] In this embodiment, a trained generative denoiser is used for image restoration, such as Figure 4 As shown, the following steps are included:

[0235] S201, initialization, the current iteration step i is initialized to the initial step N, sampling from the Gaussian distribution of the initial noise intensity to obtain the auxiliary variable

[0236] S202, the generative denoising module, uses the generative denoiser to remove the noise of the auxiliary variable u and update the clean sample estimate get

[0237] S203, data consistency module, if the current iteration step i = 1, then output the clean sample estimate In other cases, the current clean sample is estimated As the initialization of u and perform Langevin algorithm iteration until convergence:

[0238]

[0239] S204, annealing, i decreases by 1, and the current noise intensity is set to σ i If the current iteration step i = 1, then the clean sample estimate is output Otherwise, return to S202.

[0240] In this comparative example, image restoration of natural images is achieved, where the natural images come from the FFHQ dataset, and the degradation model selects the degradation operator corresponding to the Gaussian blur kernel and four-fold downsampling. For this image restoration scenario, three other image restoration methods based on diffusion models are compared, namely the DPS method proposed in "Diffusion Posterior Sampling for General Noisy Inverse Problems" published at the 2023 International Conference on Learning Representations (ICLR), the DiffPIR method proposed in "Denoising Diffusion Models for Plug-and-Play Image Restoration" published at the 2023 Conference on Computer Vision and Pattern Recognition (CVPR), and the PnP-DM method proposed in "Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors" published at the 2024 Annual Conference on Neural Information Processing Systems (NeurIPS). The evaluation criteria are peak signal-to-noise ratio (PSNR, higher is better), learned perceptual patch similarity (LPIPS, lower is better), and number of neural network calls (NFE, lower is better).

[0241] Experimental results show that under the experimental condition that the degradation operator is four times downsampling, the PSNR (27.93) of the natural image restored by the above-mentioned image restoration method of the present application is higher than the PSNR (24.38) of the DPS method, the PSNR (26.64) of the DiffPIR method, and the PSNR (27.77) of the PnP-DM method; the LPIPS of the natural image restored by the above-mentioned image restoration method of the present application is lower than the LPIPS (0.260) of the DPS method, the LPIPS (0.260) of the DiffPIR method, and the LPIPS (0.190) of the PnP-DM method; the NFE required for the system of this embodiment to restore a natural image is 50, which is less than the 1000 NFE required by the DPS method, the 100 NFE of the DiffPIR method, and the 2483 NFE of the PnP-DM method.

[0242] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDRSDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc. The above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0243] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0244] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0245] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0246] In practical applications, this computer device can be a server or a personal computer, and is not limited to basic components but can also include other hardware devices determined by functional requirements. It should be noted that the processes of this technical solution can be implemented using corresponding system modules, devices, or units. Technicians can build systems based on the technical content of these methods, and these method examples can be considered as preferred options for building systems.

[0247] It is worth noting that, in addition to using program code to implement the systems and devices provided by this technical solution, the same functions can also be implemented using hardware methods such as logic programming, logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the systems and devices provided by this technical solution can be considered hardware components, and the devices that implement different functions can be either software modules or hardware structures.

[0248] The various embodiments in this specification are described in a progressive manner, with each example primarily illustrating the differences from the other examples. Similar parts can be referenced across them. The device and computer device described herein are described briefly because they correspond to the description of the method. For details, please refer to the description of the method section.

[0249] The above describes the specific embodiments of the present invention. However, it should be understood that the present invention is not limited to these specific descriptions, and professionals in the field may make changes or modifications within the scope of their rights. These changes or modifications do not affect the core concept of the present invention.

Claims

1. A generative denoiser training method based on diffusion model distillation, characterized in that: include: Use the pre-trained diffusion model as the teacher diffusion model; Initialize the generative denoiser, scoring model and discriminator; Acquire a noise-free signal and combine it with the generative denoiser to obtain a noisy signal and Gaussian noise; Obtaining an effective noise intensity and an effective noise signal according to the noisy signal and the Gaussian noise; estimating a data score using the teacher diffusion model according to the effective noise intensity, the effective noise signal, and the noisy signal; estimating a model score using the scoring model according to the noisy signal; Calculate and optimize the gradient of the learnable parameters of the generative denoiser based on the noisy signal, the data score, the model score and the discriminator; Calculating the denoising score match of the scoring model and the adversarial loss of the discriminator based on the noisy signal, the data score, the model score, and the optimized generative denoiser, and optimizing the scoring model and the discriminator; Repeat the above steps until convergence.

2. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The generative denoiser obtained by training is μ θ (x0|y σ ), y σ Represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; using neural network Parameterized μ θ (x0|y σ ), represents the intermediate noisy signal, represents the intermediate noise intensity; θ is D θ The learnable parameters are used to initialize the neural network with the teacher diffusion model D0 Output of the neural network Satisfy x0~μ θ (x0|y σ ); The generative denoiser generates a noise-free signal based on a noisy signal, comprising: Get the first noisy signal y σ and the intensity σ of the first Gaussian noise; The noise intensity is amplified by the scaling factor γ to obtain the intermediate noise intensity. Get the standard Gaussian noise z by sampling from a standard Gaussian distribution with an expectation of 0 and a variance of 1, and scale z And add the first noisy signal y σ , get the intermediate noisy signal The first noisy signal y σ And the intensity σ of the first Gaussian noise is input into the trainable ControlNet, and the trainable ControlNet encodes the first noisy signal y through the encoder network σ with the first Gaussian noise intensity σ to preserve the intermediate noisy signal Key control information lost due to noise; According to the intermediate noisy signal And the key control information, through the teacher diffusion model D with parameter θ θ , get the denoised signal x θ .

3. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The scoring model is D φ (x t ,t,y σ ,σ),y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input, x t represents the second noise signal, t represents the noise intensity t of the second Gaussian noise, and φ is D φ The learnable parameters are initialized with the teacher diffusion model D0; The score model is used to estimate the score function of the generative denoiser, including: The first noise signal y σ and the noise intensity σ of the first Gaussian noise is input to the conditional encoder; The second noise signal x t and the noise intensity t of the second Gaussian noise is input to the backbone encoder; Combine the output of the backbone encoder and the output of the conditional encoder using a learnable amplitude-preserving addition layer; The output of the amplitude-preserving addition layer is input to a decoder to obtain an estimate of the score function of the generative denoiser.

4. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The discriminator is C ψ (x t ,t,y σ ,σ),y σ represents the first noisy signal input, σ represents the intensity of the Gaussian noise of the first noisy signal input; x t represents the second noise signal, t represents the noise intensity t of the second Gaussian noise; ψ = [ψ1, ψ2] is the learnable parameter of the discriminator; The discriminator uses a generative sample encoder and real sample encoder From the generated samples (x t ,t) and true samples (y σ ,σ) to extract features and distinguish them, including: Generate sample using encoder Extract the generated samples of the generative denoiser (x t ,t) features; Using real sample encoders Extract the real sample (y σ ,σ) features; Generate sample features and the real sample characteristics Splicing in the channel dimension to obtain splicing features; The concatenated features are sequentially passed through an average pooling layer and a linear layer, and a Sigmoid nonlinear activation function is used to output a classification probability within [0, 1].

5. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The step of obtaining a noise-free signal and combining it with the generative denoiser to obtain a noisy signal and Gaussian noise includes: Get the noise-free signal x0; The first Gaussian noise with a noise intensity of σ is applied to the noise-free signal x0, starting from a Gaussian noise with a mean of x0 and a variance of σ 2 The first noisy signal y is obtained by sampling the Gaussian distribution σ , where σ is sampled from the predefined first Gaussian noise intensity distribution; The first noisy signal y σ Through the generative denoiser μ θ (x0|y σ ), and obtain the denoised signal x θ ; For the denoised signal x θ Apply a second Gaussian noise of noise intensity t, starting from the mean x θ , the variance is t 2 The second noisy signal x is obtained by Gaussian distribution sampling t , where t is sampled from a predefined second Gaussian noise intensity distribution.

6. The method for training a generative denoiser based on diffusion model distillation according to claim 5, wherein: The obtaining, according to the noisy signal and the Gaussian noise, an effective noise intensity and an effective noise signal comprises: The effective noise intensity σ is calculated based on the noise intensity σ of the first Gaussian noise and the noise intensity t of the second Gaussian noise. eff , the effective noise intensity σ eff The square inverse of is the sum of the square inverse of the noise intensity σ of the first Gaussian noise and the square inverse of the noise intensity t of the second Gaussian noise; The effective noise intensity σ eff The square of the ratio of the noise intensity σ of the first Gaussian noise to the noise intensity t of the second Gaussian noise is the weighted combination of the first noisy signal y σ and the second noisy signal x t , to obtain the effective noise signal 7. The method for training a generative denoiser based on diffusion model distillation according to claim 5, wherein: The estimating the data score by using the teacher diffusion model according to the effective noise intensity, the effective noise signal and the noisy signal comprises: The output of the teacher diffusion model is obtained according to the effective noise intensity and the effective noise signal, and the ... t Subtract and scale with the square of the noise intensity t of the second Gaussian noise to estimate the data score s0.

8. The method for training a generative denoiser based on diffusion model distillation according to claim 5, wherein: The estimating a model score by using the score model according to the noisy signal includes: According to the first noisy signal y σ and the noise intensity σ of the first Gaussian noise, the second noisy signal x t and the noise intensity t of the second Gaussian noise to obtain the output of the score model, and the second noisy signal x t Subtract and scale by the square of the noise intensity t of the second Gaussian noise to estimate the model score s φ .

9. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The step of calculating and optimizing the gradient of the learnable parameters of the generative denoiser according to the noisy signal, the data score, the model score and the discriminator comprises: Calculate the loss function of the generative denoiser The gradient g with respect to the learnable parameters θ of the generative denoiser NCVSD ; Calculate the logarithmic gradient g of the discriminator with respect to the learnable parameters θ of the generative denoiser C ; According to the gradient g of the generative denoiser NCVSD and the gradient g of the discriminator C , gradient descent is used to optimize the learnable parameters θ of the generative denoiser.

10. The method for training a generative denoiser based on diffusion model distillation according to claim 1, wherein: The step of calculating the denoising score match of the scoring model and the adversarial loss of the discriminator based on the noisy signal, the data score, the model score, and the optimized generative denoiser, and optimizing the scoring model and the discriminator, comprises: The scoring model is trained synchronously with the model score matching during the generative denoiser optimization process: The discriminator is optimized using cross entropy loss during the optimization of the generative denoiser.

11. A method for controllable image generation, characterized in that: include: Based on pre-trained generative denoiser μ θ (x0|y σ ), given the energy function ε(x0,c) used to measure the degree of matching between a single image sample x0 and a given condition c; Distribution that is never normalized Sampling in the middle, obtaining the sample x0 that satisfies the given condition c, that is, realizing the controllable generation of the image, where q data (x0) is the data distribution.

12. The method for controllable image generation according to claim 11, characterized in that: The distribution is never normalized Medium sampling, including: Gradually reduce the noise intensity σ, and alternately use the generative denoiser to approximate the noise intensity from π σ (x0|u) is sampled and Langevin's algorithm is used to extract the value from π σ Sampling (u|x0) until the noise intensity σ is less than a given threshold; Among them, the generative denoiser is used to approximate the σ (x0|u) sampling: Among them, using Lang Zhiwan algorithm from π σ Sampling in (u|x0): π σ (x0|u) represents a distribution, u is the noise signal, x0 is the noise-free signal, that is, the noise-free signal x0 is generated from the noise signal u; π σ (u|x0) means that the noise signal u is obtained from the noise-free signal x0, and c is a given condition; Reduce the σ value; Repeat the above steps until the σ value is less than the given threshold, and output the x0 of the last step as the generated image.

13. The controllable image generation method according to claim 12, characterized in that: The generative denoiser is used to approximate the σ (x0|u) sampling, including: Calculate the effective noise intensity: Calculate the effective noise signal: from Sample x0; From q(x i-1 ∣x i ,x0) sample x i-1 ; Repeat the above steps until i=2; where x i Hidden variables introduced for multi-step sampling, σ i is the noise intensity of the latent variable, y σ Represents a noisy signal The Langevin algorithm is used to σ Sampling in (u|x0) includes: Among them, γ σ =C1(β -1 C2+σ -2 ) -1 is the adaptive step size; C1, C2, β are adjustable hyper parameters; u k To realize the intermediate iterative process of sampling u based on Langevin algorithm, if the total number of iterations is K, use u K as the sampling sample of u; σ is π σ The noise intensity of (u|x0).

14. The method for controllable image generation according to claim 11, characterized in that: The given condition c includes one or more of the following features: The text description related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the text description c and the content of image x0; The speech related to the content of image x0, in this case the energy function ε(x0,c) is defined as the degree of correlation between the speech c and the content of image x0; Semantic information that matches the content of image x0. In this case, the energy function ε(x0,c) is defined as the degree of correlation between the semantic information c and the content of image x0. The degraded image y associated with image x0: In the image deblurring scenario, the degraded image y is the image x0 blurred by the blur kernel; In the image super-resolution scenario, the degraded image y is the image after downsampling of image x0; In the image restoration scenario, the degraded image y is the image obtained by setting some pixel values of the image x0 to zero; In the Fourier phase retrieval scenario, the degraded image y is the image obtained by zero-padding, Fourier transform, and modulo operation of image x0; At this time, the energy function ε(x0,y) is defined as A represents the corresponding degradation operation in the above scenario.

15. An image processing system, characterized in that: include: Data module, obtains the image to be processed; The processing module is a plug-and-play generative denoiser obtained by the method according to any one of claims 1 to 10, which processes the image to be processed to achieve controllable generation of the image.

16. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the generative denoiser training method based on diffusion model distillation as described in any one of claims 1-10, or the controllable image generation method as described in any one of claims 11-14; it can be used to run the system as described in claim 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the generative denoiser training method based on diffusion model distillation as described in any one of claims 1-10, or the controllable image generation method as described in any one of claims 11-14; it can be used to run the system as described in claim 15.

Images