Image Restoration Method and Apparatus Based on Residual Denoising Diffusion Model

Abstract

This invention relates to the field of image processing technology, and provides an image restoration method and apparatus based on a residual denoising diffusion model. The residual denoising diffusion model introduces a residual diffusion mechanism, and estimates the residual of the current denoising step through a residual estimation network. By estimating the noise of the current denoising step, the conditional initialization of the image to be dehazed can be achieved. The forward diffusion process of the image to be dehazed is used as the starting point for denoising. This not only preserves the original content features of the image to be dehazed, but also reduces the randomness of image generation during the dehazing process, effectively removes the impact of haze on image quality, and improves the clarity and visual effect of the dehazed image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image restoration method and apparatus based on a residual denoising diffusion model. Background Technology

[0002] In real-world scenarios, smog is an optical scattering phenomenon caused by micron-sized aerosol particles suspended in the atmosphere, leading to problems such as decreased image contrast and color distortion, which seriously affects the performance of computer vision systems.

[0003] Existing deep learning-based dehazing methods primarily rely on synthetic haze datasets for training. However, due to discrepancies between the physical scattering models used in synthetic haze data and the optical characteristics of real haze data, these physical scattering models exhibit significant domain differences in practical applications. Furthermore, obtaining spatiotemporally aligned pairs of hazy and haze-free images in real-world scenes is difficult, limiting the generalization ability of supervised learning methods.

[0004] In recent years, the Denoising Diffusion Probabilistic Model (DDPM) has shown great potential in the field of image restoration. However, its iterative denoising mechanism based on Markov chains has generation randomness, which can easily lead to artifacts in the dehazing results that are not semantically consistent with the input image. Furthermore, the generation method of traditional methods that initializes with pure Gaussian noise will lose the underlying visual features of the input image. Summary of the Invention

[0005] This invention provides an image restoration method and apparatus based on a residual denoising diffusion model to address the deficiencies in the prior art.

[0006] This invention provides an image restoration method based on a residual denoising diffusion model, comprising:

[0007] Obtain the image to be dehazed;

[0008] The image to be dehazed is denoised to obtain a denoised image, and the denoised image is input into a residual denoising diffusion model to obtain a dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model.

[0009] The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image.

[0010] The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained after the preset number of iterations is used as the dehazed image.

[0011] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein the residual denoising diffusion model is trained based on the following steps:

[0012] Based on pairs of real haze-free images and synthetic haze images, the initial residual denoising diffusion model is trained using actual noise samples to obtain candidate models.

[0013] Based on clear image samples, transmittance map samples, and atmospheric light map samples corresponding to real foggy images, the candidate model is fine-tuned using actual noise samples to obtain the residual denoising diffusion model.

[0014] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein the method trains an initial residual denoising diffusion model using paired real hazy images and synthetic hazy images, applying actual noise samples, to obtain candidate models, including:

[0015] Calculate the actual residual sample between the real fog-free image and the synthesized foggy image, and determine the noise-added fog-free image sample based on the actual noise sample, the real fog-free image, and the actual residual sample;

[0016] The noisy, hazy-free image samples are respectively input into the initial residual estimation network and the initial noise estimation network in the initial residual denoising diffusion model to obtain the estimated residual samples output by the initial residual estimation network and the estimated noise samples output by the initial noise estimation network.

[0017] The estimated noise sample and the estimated residual sample are input into the initial diffusion network in the initial residual denoising diffusion model to obtain the first denoised image sample of the next denoising step output by the initial diffusion network. The first denoised image sample is then used as the noise-added, fog-free image sample and input into the initial residual estimation network and the initial noise estimation network respectively, until the preset number of iterations is reached.

[0018] Based on the actual noise sample and the estimated noise sample of each denoising step, the noise loss is calculated. Based on the actual residual sample and the estimated residual sample of each denoising step, the residual loss is calculated. Based on the noise loss and the residual loss, the initial residual denoising diffusion model is iteratively trained to obtain the candidate model.

[0019] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein the method involves fine-tuning a candidate model using real noise samples, based on clear image samples, transmittance map samples, and atmospheric light map samples corresponding to real foggy images, to obtain the residual denoising diffusion model, including:

[0020] The transmittance map sample is input into the transmittance map optimization network to obtain the optimized transmittance map for the next denoising step output by the transmittance map optimization network.

[0021] Based on the actual noise sample, the real foggy image is denoised to obtain a noisy foggy image sample, and the noisy foggy image sample is input into the candidate model to obtain the second denoised image sample of the next denoising step output by the candidate model.

[0022] The optimized transmittance map, the clear image sample, and the second denoised image sample are input into the fog density sensing network to obtain the third denoised image sample output by the fog density sensing network.

[0023] The third denoised image sample is input into the candidate model as the noisy and foggy image sample, and the optimized transmittance map is input into the transmittance map optimization network as the transmittance map sample, until the preset number of iterations is reached, to obtain the final optimized transmittance map and the final second denoised image sample.

[0024] Based on the atmospheric light map sample, the final optimized transmittance map, and the final second denoised image sample, a reconstructed fog map is generated;

[0025] Based on the real foggy image and the reconstructed fog map, the reconstruction loss is calculated, and based on the reconstruction loss, the candidate model is fine-tuned to obtain the residual denoising diffusion model.

[0026] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein the alternative model includes an alternative residual estimation network and an alternative noise estimation network, and both the alternative residual estimation network and the alternative noise estimation network are U-Nets;

[0027] The step of fine-tuning the candidate model based on the reconstruction loss to obtain the residual denoising diffusion model includes:

[0028] Calculate the first output feature of the first target upsampling layer in the candidate residual estimation network, which has a scale of 1 / 2 of the real foggy image, and the second output feature of the first target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model;

[0029] Calculate the third output feature of the second target upsampling layer in the candidate noise estimation network, which has a scale of 1 / 2 of the real foggy image, and the fourth output feature of the second target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model;

[0030] Calculate the memory loss based on the first output feature, the second output feature, the third output feature, and the fourth output feature;

[0031] Based on the memory loss and the reconstruction loss, the alternative model is fine-tuned to obtain the residual denoising diffusion model.

[0032] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein calculating memory loss based on the first output feature, the second output feature, the third output feature, and the fourth output feature includes:

[0033] Calculate the first L1 norm of the difference between the first output feature and the second output feature, and calculate the second L1 norm of the difference between the third output feature and the fourth output feature;

[0034] The memory loss is calculated based on the first L1 norm and the second L1 norm.

[0035] According to the present invention, an image restoration method based on a residual denoising diffusion model is provided, wherein adding noise to the image to be dehazed to obtain a noisy image includes:

[0036] The image to be dehazed is noise-added based on random Gaussian noise to obtain the noisy image.

[0037] The present invention also provides an image restoration device based on a residual denoising diffusion model, comprising:

[0038] The image acquisition module is used to acquire the image to be dehazed;

[0039] The image restoration module is used to add noise to the image to be dehazed to obtain a noisy image, and input the noisy image into the residual denoising diffusion model to obtain the dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model;

[0040] The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image.

[0041] The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained after the preset number of iterations is used as the dehazed image.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image restoration method based on the residual denoising diffusion model as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image restoration method based on the residual denoising diffusion model as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image restoration method based on the residual denoising diffusion model as described above.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The image restoration method and apparatus based on the residual denoising diffusion model provided by this invention introduces a residual diffusion mechanism. The residual estimation network estimates the residual of the current denoising step. By estimating the noise of the current denoising step, the conditional initialization of the image to be dehazed can be achieved. The forward diffusion process of the image to be dehazed is used as the starting point for denoising. This not only preserves the original content features of the image to be dehazed, but also reduces the randomness of image generation during the dehazing process, effectively removing the impact of haze on image quality and improving the clarity and visual effect of the dehazed image. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts of the image restoration method based on the residual denoising diffusion model provided by the present invention.

[0049] Figure 2 This is the second flowchart of the image restoration method based on the residual denoising diffusion model provided by the present invention.

[0050] Figure 3 This is a schematic diagram of the U-Net structure in the image restoration method based on the residual denoising diffusion model provided by the present invention.

[0051] Figure 4 This is a schematic diagram of the first stage training process of the residual denoising diffusion model in the image restoration method based on the residual denoising diffusion model provided by the present invention.

[0052] Figure 5 This is a schematic diagram of the second stage training process of the residual denoising diffusion model in the image restoration method based on the residual denoising diffusion model provided by the present invention.

[0053] Figure 6 This is a schematic diagram of the image restoration device based on the residual denoising diffusion model provided by the present invention;

[0054] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0055] Figure label:

[0056] 61: Image acquisition module; 62: Image dehazing module; 810: Processor; 820: Communication interface; 830: Memory; 840: Communication bus. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] Figure 1This is a flowchart illustrating an image restoration method based on a residual denoising diffusion model provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0059] S1, Obtain the image to be dehazed;

[0060] S2, add noise to the image to be dehazed to obtain a noisy image, and input the noisy image into the residual denoising diffusion model to obtain the dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model;

[0061] The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image.

[0062] The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained after the preset number of iterations is used as the dehazed image.

[0063] Specifically, the image restoration method based on the residual denoising diffusion model provided in this embodiment of the invention is executed by an image restoration device based on the residual denoising diffusion model. This device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0064] First, execute step S1 to obtain the image to be dehazed. This image refers to a real-world image collected on a foggy day that needs to be dehazed.

[0065] Then, step S2 is executed to add noise to the image to be dehazed, resulting in a noisy image. Here, as shown... Figure 2 As shown, random Gaussian noise can be used. Treating dehazed images Adding noise, i.e., adding noise to the image to be dehazed Add random Gaussian noise to obtain a noisy image. .

[0066] Add noise to the image By inputting the data into the residual denoising diffusion model, we can obtain the dehazed image corresponding to the dehazed image output by the residual denoising diffusion model. The dehazed image refers to the haze-free image corresponding to the image to be dehazed.

[0067] The residual denoising diffusion model (RDDM) employed includes a residual estimation network, a noise estimation network, and a diffusion network. Both the residual estimation network and the noise estimation network are connected to the diffusion network. The residual estimation network can utilize the noisy image... The noise estimation network estimates the residual at the current denoising step t, while the noise estimation network can use the noisy image to estimate the noise at the current denoising step t. It can be understood that the current denoising step t is the t-th noise-adding step. That is, the noise-adding steps increase sequentially, being 1, 2, 3, ..., t-1, t, while the denoising steps decrease sequentially, being t, t-1, ..., 2, 1.

[0068] This residual denoising diffusion model is based on the framework of the traditional denoising diffusion model. It decouples the diffusion process into a residual diffusion process and a noise diffusion process. The residual diffusion process is implemented through a residual estimation network, which is mainly deterministic and guides the directional diffusion of the denoised image to the image to be dehazed. The noise diffusion process is implemented through a noise estimation network, which emphasizes diversity and reflects the randomness of diffusion.

[0069] The diffusion network can use the residual and noise of the current denoising step t to calculate the mean of the denoised image in the next denoising step t-1. Based on the mean and variance of the denoised image in the next denoising step t-1, it estimates the denoised image in the next denoising step t-1. The denoised image in the next denoising step t-1 is then used as the denoised image and fed into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached. The denoised image obtained after the preset number of iterations is then used as the dehazed image. .

[0070] The forward diffusion process of this residual denoising diffusion model can be expressed as:

[0071] ;

[0072] Where J represents the generated image to be dehazed. Clear images; Image to be dehazed The residual between the clear image J and the image J It is the product of the residual weights of the t noisy steps, i.e. , It is the residual weight of the t-th noise-adding step. It is the product of the noise weights for t noise-adding steps, i.e. , It is the noise weight for the t-th noise-adding step. Since the forward diffusion process can be regarded as a Markov process, the forward diffusion process for any noise-adding step can be represented by this formula.

[0073] For the diffusion network in the residual denoising diffusion model, the noisy image Denoising image to the next denoising step The transition probability of backpropagation can be expressed as: According to Bayes' theorem, the transition probability can be written as:

[0074] ;

[0075] From the forward propagation formula We can obtain the probability distribution of forward propagation as follows:

[0076]

[0077] in, Known , and Forward propagation The probability distribution, Known and Forward propagation The probability distribution, Known and Forward propagation The probability distribution, It follows a normal distribution.

[0078] Combining the probability distribution of forward propagation The transition probability distribution can be obtained:

[0079]

[0080] in, It is the mean of the denoised image in the next denoising step t-1. is the variance of the denoised image in the next denoising step t-1, and I is a unit vector. The mean can be calculated from this variance:

[0081]

[0082] Will Substitute into the transition probability distribution and eliminate It can be obtained Since the variance consists only of known parameters and does not require prediction, it is only necessary to estimate the residuals and noise of each denoising step, and then calculate the denoised image for the next denoising step using the estimated residuals and noise of each denoising step. The mean of the denoised image is used, combined with the variance of the denoised image in the next denoising step, to obtain the denoised image of the next denoising step. Distribution .

[0083] The denoised image for the next denoising step The denoised image is input into the residual estimation network and the noise estimation network respectively, and a new round of iteration is performed to obtain the denoised image at denoising step t-2. The dehazed image can be obtained by iterating T times. Here, T is the preset denoising step, which can be set as needed. For example, it can be set to 10, that is, 10 iterations.

[0084] In this embodiment of the invention, both the residual estimation network and the noise estimation network can be made U=Net, such as... Figure 3 As shown, U-Net includes an input layer, an encoder, a decoder, and an output layer. Figure 3 The input layer consists of a 3×3 convolutional layer (Conv3×3), a batch normalization layer (Batch Norm), and Leaky ReLU. The encoder consists of four downsampled layers at different scales, and the decoder consists of four upsampled layers at different scales. The feature maps of the downsampled layers at different scales in the encoder are directly concatenated with the feature maps of the upsampled layers at the same scale in the decoder to fuse shallow and deep features, enabling the residual estimation network and noise estimation network to be better used to estimate residuals and noise. The output layer consists of a 1×1 convolutional layer (Conv1×1).

[0085] The image restoration method based on the residual denoising diffusion model provided in this embodiment of the invention introduces a residual diffusion mechanism. The residual estimation network estimates the residual of the current denoising step. By estimating the noise of the current denoising step, the conditional initialization of the image to be dehazed can be achieved. The forward diffusion process of the image to be dehazed is used as the starting point for denoising. This not only preserves the original content features of the image to be dehazed, but also reduces the randomness of image generation during the dehazing process, effectively removing the impact of haze on image quality and improving the clarity and visual effect of the dehazed image.

[0086] Based on the above embodiments, the residual denoising diffusion model is trained using the following steps:

[0087] Based on pairs of real haze-free images and synthetic haze images, the initial residual denoising diffusion model is trained using actual noise samples to obtain candidate models.

[0088] Based on clear image samples, transmittance map samples, and atmospheric light map samples corresponding to real foggy images, the candidate model is fine-tuned using actual noise samples to obtain the residual denoising diffusion model.

[0089] Specifically, the training process of the residual denoising diffusion model can be divided into two stages. In the first stage, pairs of real haze-free images and synthetic haze images, combined with actual noise samples, are used to train the initial residual denoising diffusion model to obtain a candidate model. In the second stage, real haze images and their corresponding clear image samples, transmittance map samples, and atmospheric light map samples, combined with actual noise samples, are used to fine-tune the candidate model to obtain the residual denoising diffusion model.

[0090] Here, the synthesized foggy image is obtained by synthesizing a fog map from a real fog-free image. The actual noise samples used in both stages can be the same, and can both be random Gaussian noise.

[0091] In the second stage, the clear image samples, transmittance map samples, and atmospheric light map samples corresponding to the real foggy images can be obtained by decomposing the real foggy images using an atmospheric scattering model.

[0092] In this embodiment of the invention, a two-stage training strategy is adopted: first, pre-training is performed using synthetic foggy images, and then fine-tuning is performed on real foggy images. This can reduce the training difficulty of the residual denoising diffusion model and improve training efficiency.

[0093] Based on the above embodiments, the initial residual denoising diffusion model is trained using paired real hazy images and synthetic hazy images, and actual noise samples are applied to obtain alternative models, including:

[0094] Calculate the actual residual sample between the real fog-free image and the synthesized foggy image, and add noise to the real fog-free image based on the actual noise sample to obtain a noisy fog-free image sample.

[0095] The noisy, hazy-free image samples are respectively input into the initial residual estimation network and the initial noise estimation network in the initial residual denoising diffusion model to obtain the estimated residual samples output by the initial residual estimation network and the estimated noise samples output by the initial noise estimation network.

[0096] The estimated noise sample and the estimated residual sample are input into the initial diffusion network in the initial residual denoising diffusion model to obtain the first denoised image sample of the next denoising step output by the initial diffusion network. The first denoised image sample is then used as the noise-added, fog-free image sample and input into the initial residual estimation network and the initial noise estimation network respectively, until the preset number of iterations is reached.

[0097] Based on the actual noise sample and the estimated noise sample of each denoising step, the noise loss is calculated. Based on the actual residual sample and the estimated residual sample of each denoising step, the residual loss is calculated. Based on the noise loss and the residual loss, the initial residual denoising diffusion model is iteratively trained to obtain the candidate model.

[0098] Specifically, during the first phase of training, such as Figure 4 As shown, this can be achieved by synthesizing a foggy image. Subtracting the actual residual sample from the true haze-free image GT yields the actual residual sample. And using actual noise samples, that is, using random Gaussian noise. and real haze-free image GT and actual residual samples By combining the forward diffusion process, noisy and hazy image samples were identified. .

[0099] Subsequently, the noisy, haze-free image samples were... The initial residual estimation network and the initial noise estimation network in the initial residual denoising and diffusion model are respectively input to obtain the estimated residual samples output by the initial residual estimation network. The estimated noise samples output by the initial noise estimation network .in, θ These are the model parameters for the initial residual denoising diffusion model.

[0100] The estimated noise sample and the estimated residual sample are input into the initial diffusion network in the initial residual denoising diffusion model to obtain the first denoised image sample of the next denoising step output by the initial diffusion network. The first denoised image sample is then used as a noise-added, fog-free image sample and input into the initial residual estimation network and the initial noise estimation network respectively, until the preset number of iterations is reached.

[0101] The noise loss is calculated using the expected value of the norm square of the difference between the actual noise sample and the estimated noise sample at each denoising step. The residual loss is calculated using the expected value of the norm square of the difference between the actual residual sample and the estimated residual sample at each denoising step. That is:

[0102] ;

[0103] in, To estimate the weights corresponding to the residual samples, To estimate the weights corresponding to the noisy samples, Represents the norm.

[0104] Subsequently, the first-stage training loss can be obtained by weighted summing of the noise loss and the residual loss, i.e.:

[0105] ;

[0106] in, All of these are hyperparameters. This represents the losses from the first phase of training.

[0107] Finally, using the training loss from the first stage, the initial residual estimation network and the initial noise estimation network can be iteratively trained to obtain the candidate model.

[0108] Based on the above embodiments, the step of fine-tuning the candidate model using clear image samples, transmittance map samples, and atmospheric light map samples corresponding to real foggy images, and applying actual noise samples, to obtain the residual denoising diffusion model includes:

[0109] The transmittance map sample is input into the transmittance map optimization network to obtain the optimized transmittance map for the next denoising step output by the transmittance map optimization network.

[0110] Based on the actual noise sample, the real foggy image is denoised to obtain a noisy foggy image sample, and the noisy foggy image sample is input into the candidate model to obtain the second denoised image sample of the next denoising step output by the candidate model.

[0111] The optimized transmittance map, the clear image sample, and the second denoised image sample are input into the fog density sensing network to obtain the third denoised image sample output by the fog density sensing network.

[0112] The third denoised image sample is used as the denoised and foggy image sample and input into the candidate model, and the optimized transmittance map is used as the transmittance map sample and input into the transmittance map optimization network, until the preset number of iterations is reached;

[0113] Based on the atmospheric light map sample, the optimized transmittance map with the preset number of iterations, and the second denoised image sample, a reconstructed fog map is generated;

[0114] Based on the real foggy image and the reconstructed fog map, the reconstruction loss is calculated, and based on the reconstruction loss, the candidate model is fine-tuned to obtain the residual denoising diffusion model.

[0115] Specifically, during the second phase of training, such as Figure 5 As shown, clear image samples correspond to real foggy images. Transmittance map sample And atmospheric light image sample A.

[0116] Transmittance map sample The data is input into the transmittance map optimization network to obtain the optimized transmittance map for the next denoising step, which is output by the network. The transmittance map optimization network can also be a U-Net.

[0117] By using actual noise samples, noise is added to real foggy images to obtain noisy foggy image samples. and add noise to the foggy image samples The input is fed into the candidate model to obtain the second denoised image sample for the next denoising step, which is output by the candidate model. .

[0118] Optimize the transmittance map Clear image samples and the second denoised image sample The input is fed into the fog density sensing network to obtain the third denoised image sample output by the fog density sensing network. And there are:

[0119] .

[0120] Subsequently, the third denoised image sample The noisy, hazy image samples are input into the candidate model, and the transmittance map is optimized. The transmittance map samples are input into the transmittance map optimization network until the preset number of iterations T is reached, resulting in the final optimized transmittance map and the final second denoised image sample.

[0121] Using atmospheric light map sample A, the final optimized transmittance map and the final second denoised image sample By combining atmospheric scattering models, a reconstructed fog map is generated. That is:

[0122] .

[0123] Subsequently, using real foggy images and reconstructing fog maps Calculate reconstruction loss That is: ,in, It is an L2 norm.

[0124] Finally, the reconstruction loss can be directly used as the training loss in the second stage to iteratively fine-tune the structural parameters of the candidate model, thus obtaining the residual denoising diffusion model.

[0125] In this embodiment of the invention, when calculating the reconstruction loss, a fog density sensing network is introduced, and the transmittance map is used as a mask to focus on processing distant foggy areas that are more difficult to defog, thereby improving the defogging accuracy of the trained residual denoising diffusion model.

[0126] Based on the above embodiments, the alternative model includes an alternative residual estimation network and an alternative noise estimation network, both of which are U-Nets;

[0127] The step of fine-tuning the candidate model based on the reconstruction loss to obtain the residual denoising diffusion model includes:

[0128] Calculate the first output feature of the first target upsampling layer in the candidate residual estimation network, which has a scale of 1 / 2 of the real foggy image, and the second output feature of the first target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model;

[0129] Calculate the third output feature of the second target upsampling layer in the candidate noise estimation network, which has a scale of 1 / 2 of the real foggy image, and the fourth output feature of the second target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model;

[0130] Calculate the memory loss based on the first output feature, the second output feature, the third output feature, and the fourth output feature;

[0131] Based on the memory loss and the reconstruction loss, the alternative model is fine-tuned to obtain the residual denoising diffusion model.

[0132] Specifically, to avoid the model losing the knowledge gained during training on the synthetic dataset, a memory loss is introduced in the second-stage training loss. This memory loss can preserve the candidate model, so that, under the same input, the residual and noise features extracted in the fine-tuned residual estimation network and the fine-tuned noise estimation network of the model after fine-tuning the candidate model can be as consistent as possible with the features extracted from the candidate model.

[0133] Memory loss can be determined through the following steps:

[0134] The first output feature of the first target upsampling layer in the candidate residual estimation network, which is scaled to half the size of the real foggy image, is calculated. The second output feature of the first target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model is also calculated. Here, the candidate residual estimation network refers to the residual estimation network in the candidate model, and the fine-tuned residual estimation network refers to the residual estimation network in each fine-tuned model obtained when fine-tuning the candidate model. The first target upsampling layer is the penultimate upsampling layer in the candidate residual estimation network, and the second target upsampling layer is the penultimate upsampling layer in the fine-tuned residual estimation network.

[0135] The third output feature of the second target upsampling layer in the candidate noise estimation network, which is scaled to half the size of the real foggy image, is calculated. The fourth output feature of the second target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model is also calculated. Here, the candidate noise estimation network refers to the noise estimation network in the candidate model, and the fine-tuned noise estimation network refers to the noise estimation network in each fine-tuned model obtained when fine-tuning the candidate model. The second target upsampling layer is the penultimate upsampling layer in the candidate noise estimation network, and the second target upsampling layer is the penultimate upsampling layer in the fine-tuned noise estimation network.

[0136] The memory loss can be calculated using the first output feature, the second output feature, the third output feature, and the fourth output feature.

[0137] For example, the first L1 norm of the difference between the first and second output features can be calculated, and the second L1 norm of the difference between the third and fourth output features can be calculated. Then, the first and second L1 norms are added together to obtain the memory loss. That is:

[0138] ;

[0139] in, Indicates the third output feature; Indicates the fourth output feature; Indicates the first output feature; This represents the second output feature. It is an L1 norm. This is due to memory loss.

[0140] Subsequently, the memory loss and reconstruction loss can be weighted and summed to obtain the second-stage training loss. That is:

[0141] ;

[0142] in, All of these are hyperparameters. This represents a loss during the second phase of training.

[0143] In this embodiment of the invention, since the output features of the penultimate upsampling layer of the residual estimation network and noise estimation network in both the candidate model and the fine-tuned model incorporate multi-level, multi-scale, and cross-channel information, the memory loss can be calculated to allow the final residual denoising diffusion model to retain the knowledge already acquired by the candidate model. Introducing memory loss into the second-stage training loss can reduce the difference between synthetic foggy images and real foggy images, enhancing the domain adaptability of the residual denoising diffusion model.

[0144] In summary, the image restoration method based on a residual denoising diffusion model provided in this embodiment of the invention has significant advantages over traditional methods in terms of dehazing effect, model generalization ability, and image detail preservation. This method not only preserves the original content features of the image to be dehazed, such as details and colors, but also focuses on processing more difficult-to-dehaze areas during the dehazing process, thereby achieving more accurate image restoration and providing a more reliable and efficient solution for the application of computer vision systems in real-world scenarios.

[0145] like Figure 6 As shown, based on the above embodiments, this embodiment of the invention provides an image restoration device based on a residual denoising diffusion model, comprising:

[0146] Image acquisition module 61 is used to acquire the image to be dehazed;

[0147] Image restoration module 62 is used to add noise to the image to be dehazed to obtain a noisy image, and input the noisy image into a residual denoising diffusion model to obtain a dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model;

[0148] The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image.

[0149] The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained after the preset number of iterations is used as the dehazed image.

[0150] Specifically, the functions of each module in the image restoration device based on the residual denoising diffusion model provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0151] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the image restoration method based on the residual denoising diffusion model provided in the above embodiments.

[0152] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image restoration method based on the residual denoising diffusion model provided in the above embodiments.

[0154] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image restoration method based on the residual denoising diffusion model provided in the above embodiments.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image restoration method based on a residual denoising diffusion model, characterized in that, include: Obtain the image to be dehazed; The image to be dehazed is denoised to obtain a denoised image, and the denoised image is input into a residual denoising diffusion model to obtain a dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model. The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image. The residual denoising diffusion model decouples the diffusion process into a residual diffusion process and a noise diffusion process. The residual diffusion process is implemented by the residual estimation network to guide the directional diffusion of the denoised image to the image to be dehazed, and the noise diffusion process is implemented by the noise estimation network. The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained by the preset number of iterations is used as the dehazing image; The forward diffusion process of the residual denoising diffusion model is expressed as follows: ， in, J represents the image to be dehazed, where J is the image to be generated. Clear images; Image to be dehazed The residual between the clear image J and the image J It is the product of the residual weights of the t noisy steps, i.e. , It is the residual weight of the t-th noise-adding step. It is the product of the noise weights for t noise-adding steps, i.e. , It is the noise weight for the t-th noise-adding step. It is to obtain a noisy image. Treating dehazed images Added Gaussian noise; The residual denoising diffusion model is trained in two stages. In the first stage, the initial residual denoising diffusion model is trained using pairs of real hazy images and synthetic hazy images, combined with actual noise samples, to obtain a candidate model. In the second stage, the candidate model is fine-tuned using real hazy images and their corresponding clear image samples, transmittance map samples, and atmospheric light map samples, combined with actual noise samples, to obtain the residual denoising diffusion model. The training loss in the first stage is obtained by weighted summation of noise loss and residual loss, as shown below: ， in, All of these are hyperparameters. This represents the training loss in the first phase. For noise loss, This is the residual loss; The second phase of training loss is the reconstruction loss; The calculation formulas for the noise loss and the residual loss are as follows: ， in, To estimate the weights corresponding to the residual samples, To estimate the weights corresponding to the noisy samples, Represents the norm, For actual residual samples, To estimate the residual sample, For noisy, hazy-free image samples, To estimate the noise sample, As expected.

2. The image restoration method based on the residual denoising diffusion model according to claim 1, characterized in that, Based on pairs of real haze-free images and synthetic haze images, and using actual noise samples, the initial residual denoising diffusion model is trained to obtain candidate models, including: Calculate the actual residual sample between the real fog-free image and the synthesized foggy image, and determine the noise-added fog-free image sample based on the actual noise sample, the real fog-free image, and the actual residual sample; The noisy, hazy-free image samples are respectively input into the initial residual estimation network and the initial noise estimation network in the initial residual denoising diffusion model to obtain the estimated residual samples output by the initial residual estimation network and the estimated noise samples output by the initial noise estimation network. The estimated noise sample and the estimated residual sample are input into the initial diffusion network in the initial residual denoising diffusion model to obtain the first denoised image sample of the next denoising step output by the initial diffusion network. The first denoised image sample is then used as the noise-added, fog-free image sample and input into the initial residual estimation network and the initial noise estimation network respectively, until the preset number of iterations is reached. Based on the actual noise sample and the estimated noise sample of each denoising step, the noise loss is calculated. Based on the actual residual sample and the estimated residual sample of each denoising step, the residual loss is calculated. Based on the noise loss and the residual loss, the initial residual denoising diffusion model is iteratively trained to obtain the candidate model.

3. The image restoration method based on the residual denoising diffusion model according to claim 1, characterized in that, Based on clear image samples, transmittance map samples, and atmospheric light map samples corresponding to real foggy images, and applying actual noise samples, the candidate model is fine-tuned to obtain the residual denoising diffusion model, including: The transmittance map sample is input into the transmittance map optimization network to obtain the optimized transmittance map for the next denoising step output by the transmittance map optimization network. Based on the actual noise sample, the real foggy image is denoised to obtain a noisy foggy image sample, and the noisy foggy image sample is input into the candidate model to obtain the second denoised image sample of the next denoising step output by the candidate model. The optimized transmittance map, the clear image sample, and the second denoised image sample are input into the fog density sensing network to obtain the third denoised image sample output by the fog density sensing network. The third denoised image sample is input into the candidate model as the noisy and foggy image sample, and the optimized transmittance map is input into the transmittance map optimization network as the transmittance map sample, until the preset number of iterations is reached, to obtain the final optimized transmittance map and the final second denoised image sample. Based on the atmospheric light map sample, the final optimized transmittance map, and the final second denoised image sample, a reconstructed fog map is generated; Based on the real foggy image and the reconstructed fog map, the reconstruction loss is calculated, and based on the reconstruction loss, the candidate model is fine-tuned to obtain the residual denoising diffusion model.

4. The image restoration method based on the residual denoising diffusion model according to claim 3, characterized in that, The alternative model includes an alternative residual estimation network and an alternative noise estimation network, both of which are U-Nets; The step of fine-tuning the candidate model based on the reconstruction loss to obtain the residual denoising diffusion model includes: Calculate the first output feature of the first target upsampling layer in the candidate residual estimation network, which has a scale of 1 / 2 of the real foggy image, and the second output feature of the first target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model; Calculate the third output feature of the second target upsampling layer in the candidate noise estimation network, which has a scale of 1 / 2 of the real foggy image, and the fourth output feature of the second target upsampling layer in each fine-tuned residual estimation network obtained when fine-tuning the candidate model; Calculate the memory loss based on the first output feature, the second output feature, the third output feature, and the fourth output feature; Based on the memory loss and the reconstruction loss, the alternative model is fine-tuned to obtain the residual denoising diffusion model.

5. The image restoration method based on the residual denoising diffusion model according to claim 4, characterized in that, The calculation of memory loss based on the first output feature, the second output feature, the third output feature, and the fourth output feature includes: Calculate the first L1 norm of the difference between the first output feature and the second output feature, and calculate the second L1 norm of the difference between the third output feature and the fourth output feature; The memory loss is calculated based on the first L1 norm and the second L1 norm.

6. An image restoration device based on a residual denoising diffusion model, characterized in that, include: The image acquisition module is used to acquire the image to be dehazed; The image restoration module is used to add noise to the image to be dehazed to obtain a noisy image, and input the noisy image into the residual denoising diffusion model to obtain the dehazed image corresponding to the image to be dehazed output by the residual denoising diffusion model; The residual denoising diffusion model includes a residual estimation network, a noise estimation network, and a diffusion network. The residual estimation network is used to estimate the residual of the current denoising step based on the noisy image, and the noise estimation network is used to estimate the noise of the current denoising step based on the noisy image. The residual denoising diffusion model decouples the diffusion process into a residual diffusion process and a noise diffusion process. The residual diffusion process is implemented by the residual estimation network to guide the directional diffusion of the denoised image to the image to be dehazed, and the noise diffusion process is implemented by the noise estimation network. The diffusion network is used to calculate the mean of the denoised image in the next denoising step based on the residual and noise of the current denoising step, estimate the denoised image in the next denoising step based on the mean and variance of the denoised image in the next denoising step, and input the denoised image in the next denoising step as the denoised image into the residual estimation network and the noise estimation network respectively for iteration until a preset number of iterations is reached, and the denoised image obtained by the preset number of iterations is used as the dehazing image; The forward diffusion process of the residual denoising diffusion model is expressed as follows: ， in, J represents the image to be dehazed, where J is the image to be generated. Clear images; Image to be dehazed The residual between the clear image J and the image J It is the product of the residual weights of the t noisy steps, i.e. , It is the residual weight of the t-th noise-adding step. It is the product of the noise weights for t noise-adding steps, i.e. , It is the noise weight for the t-th noise-adding step. It is to obtain a noisy image. Treating dehazed images Added Gaussian noise; The residual denoising diffusion model is trained in two stages. In the first stage, the initial residual denoising diffusion model is trained using pairs of real hazy images and synthetic hazy images, combined with actual noise samples, to obtain a candidate model. In the second stage, the candidate model is fine-tuned using real hazy images and their corresponding clear image samples, transmittance map samples, and atmospheric light map samples, combined with actual noise samples, to obtain the residual denoising diffusion model. The training loss in the first stage is obtained by weighted summation of noise loss and residual loss, as shown below: ， in, All of these are hyperparameters. This represents the training loss in the first phase. For noise loss, This is the residual loss; The second phase of training loss is the reconstruction loss; The calculation formulas for the noise loss and the residual loss are as follows: ， in, To estimate the weights corresponding to the residual samples, To estimate the weights corresponding to the noisy samples, Represents the norm, For actual residual samples, To estimate the residual sample, For noisy, hazy-free image samples, To estimate the noise sample, As expected.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image restoration method based on the residual denoising diffusion model as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image restoration method based on the residual denoising diffusion model as described in any one of claims 1-5.

Images