Image defogging and fog increasing method based on residual bidirectional diffusion model

Abstract

The invention provides an image defogging and fog increasing method based on a residual bidirectional diffusion model, and the method comprises the steps: moving a window for a fogless image and a foggy image at a stride in horizontal and vertical dimensions, and decomposing the fogless image and the foggy image into a group of overlapped block dictionaries; the method comprises the following steps: carrying out noise adding processing on an image through a forward process of image defogging and fog increasing, so that an initial state in defogging approaches the distribution of a fogless image and a final state approaches the distribution of a foggy image, and an initial state in fog increasing approaches the distribution of a foggy image and a final state approaches the distribution of the fogless image; and obtaining a mean value and a variance of a reverse process through a unified deep neural network so as to restore a noise-added image.

Description

Technical Field

[0001] The invention relates to an image processing technology, in particular to an image defogging and fogging method based on a residual bidirectional diffusion model. Background Art

[0002] Photos taken on foggy days usually show low contrast and fidelity due to absorption and scattering of light when it propagates under adverse weather conditions. The process of enhancing the contrast and color integrity of images taken in a hazy environment is called dehazing. Diffusion models have demonstrated strong capabilities in modeling data distribution. Several diffusion models have achieved unprecedented success in image generation, showing potential in solving downstream low-level vision tasks. However, these models can only perform single image dehazing and find it difficult to effectively learn the mapping between haze-free images and hazy images, resulting in poor dehazing performance because they fail to capture the characteristics of fog. Although existing diffusion-based dehazing models use hazy images as priors and utilize jump connections to use information in hazy images, how to conduct a more efficient and interpretable forward process by starting from a prior distribution based on hazy images still needs further exploration. Summary of the invention

[0003] The object of the present invention is to provide an image defogging and fogging method based on a residual bidirectional diffusion model, comprising:

[0004] Step S100, for the haze-free image x0 and the haze-containing image y0, a window of size p×p is moved with a stride r in the horizontal and vertical dimensions to decompose the haze-free image x0 and the haze-containing image y0 into a set of overlapping block dictionaries;

[0005] Step S200, performing noise processing on the image through the forward process of image defogging and fogging, so that the initial state in defogging is close to the distribution of the fog-free image and the final state is close to the distribution of the foggy image, and the initial state in fogging is close to the distribution of the foggy image, and the final state is close to the distribution of the fog-free image;

[0006] Step S300, by a Deep Neural Network The mean and variance of the inverse process are obtained to restore the noisy image.

[0007] Furthermore, in step S100, a binary mask matrix P with the same dimensions as the foggy image x0 and the fog-free image y0 is randomly generated. i , by setting and , extract a patch located at p×p from a training image pair (x0, y0), where Operation from Extract the block at the indicated location.

[0008] Furthermore, in step S200, the forward process of the image defogging process includes:

[0009] Step S201, obtaining the i-th block dictionary of the foggy image and the dictionary of the ith block of the haze-free image The residual between ,Right now and The difference between the pixel values ​​​​between;

[0010] Step S202: Establish a Markov chain with a length of T

[0011] (1)

[0012] Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the adjustment parameter, I is the unit matrix;

[0013] Step S203, from t x = 0 and t x =T, according to formula (2), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the i-th block of the haze-free image Get the dictionary of the i-th block of the foggy image The approximate distribution of

[0014] . (2)

[0015] Furthermore, the forward process of image fogging in step S200 includes:

[0016] Step S211, obtaining the i-th block dictionary of the fog-free image and the dictionary of the i-th block of the foggy image The residual between ,Right now and The difference between the pixel values ​​​​between;

[0017] Step S212: Establish a Markov chain with a length of T

[0018] (3)

[0019] Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the scaling factor, I is the identity matrix;

[0020] Step S213, according to formula (3), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the i-th block of the foggy image Get the dictionary of the i-th block of the haze-free image The approximate distribution of

[0021] . (4)

[0022] Further, step S300 includes:

[0023] Step S301, setting optimization parameters Minimize the negative evidence lower bound of

[0024] (6)

[0025] in, Represents KL divergence calculation; target distribution and According to formula (5), the processing is explicitly expressed as

[0026] (7)

[0027] According to Bayes' theorem

[0028] (8)

[0029] Step S302: Obtain the predicted variance of the noise

[0030] (9)

[0031] Step S303, obtaining the mean value of noise prediction

[0032] (10)

[0033] in, is a of deep neural networks.

[0034] Furthermore, according to formula (10), the objective function of formula (6) is simplified to

[0035] (11)

[0036] in, is the weight.

[0037] Further, .

[0038] Compared with the prior art, the present invention has the following advantages: (1) A dual Markov chain is designed to effectively adjust the residuals and promote bidirectional smooth conversion between them; (2) The foggy and fog-free images are perturbed by injecting noise at different time steps, and the noise in the perturbed data is predicted, thereby simultaneously learning the conditional distribution; (3) In order to improve the performance and reduce the computational cost on a relatively small dataset, a unified scoring function is introduced that is learned based on image blocks rather than the entire image. With only 15 sampling steps, bidirectional conversion between foggy and fog-free images of different sizes is successfully achieved.

[0039] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0041] Figure 2 This is a schematic diagram of the visualization results of the present invention and other methods on the RESIDE-6K dataset.

[0042] Figure 3Schematic diagram of the visualization results of the present invention and other methods on the NTIRE2020, NTIRE2021 and NTIRE2023 datasets.

[0043] Figure 4 Schematic diagram of effectiveness evaluation of the present invention, wherein sub-images (c) and (f) are haze feature images corresponding to the foggy input image (a) and the haze output image (d) generated from the haze-free reference image (b), sub-image (h) compares the average RGB channel intensity curves of the haze-free reference image (b) and the dehazed output (e), and sub-image (g) is a haze distribution difference map between the initial foggy input and the model's haze output for the haze-free image. DETAILED DESCRIPTION

[0044] Combination Figure 1 , an image defogging and fogging method based on residual bidirectional diffusion model (RBDM), comprising:

[0045] Step S100: for the fog-free image x0 and the foggy image y0, a binary mask matrix P with the same dimension as the foggy image x0 and the fog-free image y0 is randomly generated. i , represents the position of the i-th p×p block in the image; for the haze-free image x0 and the haze-covered image y0, a window of size p×p is moved with a stride r in the horizontal and vertical dimensions to decompose the haze-free image x0 and the haze-covered image y0 into a set of overlapping block dictionaries;

[0046] Step S200, performing noise processing on the image through the forward process of image defogging and fogging, so that the initial state in defogging is close to the distribution of the fog-free image and the final state is close to the distribution of the foggy image, and the initial state in fogging is close to the distribution of the foggy image, and the final state is close to the distribution of the fog-free image;

[0047] Step S300, by a Deep Neural Network The mean and variance of the inverse process are obtained to restore the noisy image.

[0048] In step S100, by setting and , extract a patch located at p×p from a training image pair (x0, y0), where Operation from Extract the block at the indicated location.

[0049] In step S200, the forward process of the image defogging process includes:

[0050] Step S201, obtaining the i-th block dictionary of the foggy image and the dictionary of the ith block of the haze-free image The residual between ,Right now and The difference between the pixel values ​​​​between;

[0051] Step S202: Establish a Markov chain with a length of T

[0052] (1)

[0053] Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the adjustment parameter, I is the unit matrix;

[0054] Step S203, from t x = 0 and t x =T, according to formula (2), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the i-th block of the haze-free image Get the dictionary of the i-th block of the foggy image The approximate distribution of

[0055] . (2)

[0056] control The transition speed from the initial state to the final state, and and ; By parameter Adjust the variance by make sure and Since formula (1) has a smooth transition between x The integral of the marginal distribution of can be resolved as , so x1 and x T The marginal distributions of and By setting the transition distribution according to formula (1), the initial state in the defogging task approximates the distribution of the haze-free image, while the final state approximates the distribution of the haze-free image. This enables the model to effectively learn to reconstruct the haze-free image from the haze-free image.

[0057] The forward process of image fogging in step S200 includes:

[0058] Step S211, obtaining the i-th block dictionary of the fog-free image and the dictionary of the i-th block of the foggy image The residual between ,Right now and The difference between the pixel values ​​​​between;

[0059] Step S212: Establish a Markov chain with a length of T

[0060] (3)

[0061] Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the scaling factor, I is the identity matrix;

[0062] Step S213, according to formula (3), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the i-th block of the foggy image Get the dictionary of the i-th block of the haze-free image The approximate distribution of

[0063] . (4)

[0064] control The transition speed from the initial state to the final state, and and ; By parameter Adjust the variance by make sure and Since formula (3) has a smooth transition between y The integral of the marginal distribution of can be resolved as , so y1 and y T The marginal distributions of and By setting the transition distribution according to formula (3), the initial state approaches the distribution of foggy images, while the final state approaches the distribution of fog-free images. This enables the model to effectively synthesize foggy images from fog-free images.

[0065] The reverse process in step S300 is to restore the noisy image to make it clear, aiming to estimate the posterior distribution and , get the mean and variance. Corresponding to all t x and t y have

[0066] (5)

[0067] in, , is from arrive The inverse transformation kernel has learnable parameters ; , is from arrive This embodiment does not distinguish between the defogging and fogging process networks, but uses a unified network with parameters The deep neural network covers two scenarios. The specific principles and processes are as follows:

[0068] Step S301, setting optimization parameters Minimize the negative evidence lower bound of

[0069] (6)

[0070] in, Represents KL divergence calculation; target distribution and According to formula (5), the processing is explicitly expressed as

[0071] (7)

[0072] According to Bayes' theorem

[0073] (8)

[0074] Step S302, obtain the predicted variance of the noise, considering the variance parameter and and No matter, get

[0075] (9)

[0076] Step S303, obtaining the mean value of noise prediction

[0077] (10)

[0078] in, is a According to formula (10), the objective function of formula (6) is simplified to

[0079] (11)

[0080] Among them, when can significantly improve performance.

[0081] At each sampling time step t, use the noise estimation network Directly estimate the clean image of each block, accumulate these overlapping results into a matrix Sample of the same size as the entire image, and record the overlapping area using a mask matrix M of the same size. At the end of the current sampling time step, the overlapping pixel parts are averaged according to the mask to obtain the final output image.

[0082] Example

[0083] This example is based on the public datasets RESIDE-6K, NTIRE2020 ] , NTIRE2021, and NTIRE2023. RESIDE-6K contains 3000 pairs of indoor synthetic images and 3000 pairs of outdoor images, of which 5000 pairs are used for training and 1000 pairs are used for testing. The NTIRE2020, NTIRE2021, and NTIRE2023 datasets show non-uniformly distributed haze patterns. Specifically, NTIRE2020 contains 45 pairs of training images, 5 pairs of validation images, and 5 pairs of test images. NTIRE2021 contains 25 pairs of training images, 5 pairs of validation images, and 5 pairs of test images. NTIRE2023 contains 40 pairs of training images, 5 pairs of validation images, and 5 pairs of test images.

[0084] This embodiment is implemented using PyTorch 1.13.1 and trained on a PC configured with an Intel(R) Core(TM) i5-13600KCPU @5.10GHz and an NVIDIA GeForce RTX 4090 GPU. The optimizer used is Adam, with a fixed learning rate of 5e-5. The network architecture is based on the UNet model in DDPM and has been modified to suit specific tasks. The RBDM of this embodiment was trained on the RESIDE-6K dataset for 190,000 iterations. Specifically, 16 images were initially extracted from the training set, and 16 64×64 blocks were randomly extracted from each image to form a small batch of 256 blocks. For NTIRE2020, NTIRE2021, and NTIRE2023 datasets, 45 pairs of training images and 5 pairs of validation images are selected from NTIRE2020, 25 pairs of training images and 5 pairs of validation images are selected from NTIRE2021, and 40 pairs of training images and 5 pairs of validation images are selected from NTIRE2023 to form the training set. The test set includes 5 pairs of test images in NTIRE2020, the last 5 pairs of training images in NTIRE2021, and the last 5 pairs of training images in NTIRE2023. 8 pairs of images are randomly selected from the training set, each image is cropped into 8 128×128 blocks, thus forming a small batch of 64 blocks, and the model is trained on this dataset for a total of 900,000 iterations.

[0085] Table 1 Quantitative comparison on the RESIDE-6K dataset. The best results are in bold and the second best results are in Underline express.

[0086] Table 1

[0087] Results on the RESIDE-6K dataset: To demonstrate the effectiveness of RBDM in image dehazing, it is compared with the latest dehazing methods, as shown in Table 1. The baseline methods include DCP based on priors, AOD-Net based on atmospheric optical model, and various convolutional neural network methods such as FFA-Net. [ and the Transformer-based method DehazeFormer. The hybrid frameworks MixDehazeNet and TSNet are included in the comparison. RBDM outperforms these methods in terms of both PSNR and SSIM, achieving scores of 32.78 and 0.976, respectively. This shows that RBDM not only performs well in dehazing, but also excels in preserving image details and textures. Figure 2Visual restoration images are shown in comparison with cutting-edge methods. It can be observed that traditional dehazing methods usually over-enhance colors, resulting in unnatural results. In contrast, RBDM is able to effectively preserve the natural colors of the image while removing haze, thus avoiding artifacts caused by over-processing. RBDM strikes a good balance between dehazing effect, detail preservation, and natural color rendering. It is able to effectively remove haze while preserving image details and natural appearance, resulting in more realistic dehazed images.

[0088] Table 2 Quantitative comparison on the NHIRE dataset. The best results are in bold and the second best results are underlined.

[0089] Table 2

[0090]

[0091] The results on the NTIRE2020 / 2021 / 2023 datasets are shown in Table 2. The RBDM of this embodiment achieves the highest PSNR and SSIM scores on the NTIRE2020, NTIRE2021, and NTIRE2023 datasets. For example, in the NTIRE2020 dataset, SCANet achieves a PSNR score of 19.52 and a SSIM score of 0.6488, while our RBDM improves the PSNR and SSIM by 3.67 dB and 0.0177, respectively. Figure 3 The RBDM method is visually compared with other state-of-the-art dehazing methods. It is worth noting that the NTIRE dataset is high-resolution, with a resolution of 1600×1200 pixels for NTIRE2020 and NTIRE2021 and 4000×6000 pixels for NTIRE2023. RBDM performs better in recovering details in light haze conditions and produces clearer and more natural images that are closer to the original images, thereby improving PSNR and SSIM scores. However, its performance degrades in heavy haze conditions, as details are usually masked by haze and difficult to recover effectively. In this case, those methods with lower PSNR and SSIM scores may still produce reasonable results in dense haze areas due to their more aggressive haze processing techniques. RBDM performs well in the high-resolution image dehazing task, especially in light haze areas, where it is able to effectively recover details and maintain color accuracy close to the original images. However, we note that there is still room for improvement in heavy haze conditions, and future work will focus on enhancing the robustness of the model under various haze densities.

[0092] pass Figure 4The presented visual analysis demonstrates the effectiveness of the proposed method. Sub-figures (c) and (f) show haze feature maps corresponding to the hazy input image (a) and the hazed output image (d) generated from the haze-free reference image (b). In these figures, brightness represents haze concentration, with higher brightness indicating greater haze concentration. Sub-figure (g) highlights the difference in haze distribution between the initial hazy input and the model’s hazed output for the haze-free image, demonstrating the model’s ability to generate realistic, non-uniform haze patterns. This result suggests that the RBDM approach can be applied to dataset augmentation for both hazy and haze-free images. Sub-figure (h) compares the average RGB channel intensity curves of the haze-free image reference (b) and the dehazed output (e). The close alignment between the curves validates the effectiveness of our method in preserving color fidelity, indicating that the dehazed image retains its natural appearance without distorting the color. Maintaining color consistency is critical for applications that require high visual accuracy, such as remote sensing, autonomous driving, and surveillance. Figure 4 The results show that RBDM has powerful capabilities in image dehazing and haze generation.

Claims

1. A method for image defogging and fog enhancement based on a residual bidirectional diffusion model, characterized in that: include: Step S100, for the haze-free image x0 and the haze-containing image y0, a window of size p×p is moved with a stride r in the horizontal and vertical dimensions to decompose the haze-free image x0 and the haze-containing image y0 into a set of overlapping block dictionaries; Step S200, performing noise processing on the image through the forward process of image defogging and fogging, so that the initial state in defogging is close to the distribution of the fog-free image and the final state is close to the distribution of the foggy image, and the initial state in fogging is close to the distribution of the foggy image, and the final state is close to the distribution of the fog-free image; Step S300, by a Deep Neural Network The mean and variance of the inverse process are obtained to restore the noisy image.

2. The method according to claim 1, characterized in that In step S100, a binary mask matrix P with the same dimensions as the foggy image x0 and the fog-free image y0 is randomly generated. i , by setting and , extract a patch located at p×p from a training image pair (x0, y0), where Operation from Extract the block at the indicated location.

3. The method according to claim 2, characterized in that In step S200, the forward process of the image defogging process includes: Step S201, obtaining the i-th block dictionary of the foggy image and the dictionary of the ith block of the haze-free image The residual between ,Right now and The difference between the pixel values ​​​​between; Step S202: Establish a Markov chain with a length of T （1） Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, and T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the adjustment parameter, I is the unit matrix; Step S203, from t x = 0 and t x =T, according to formula (2), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the i-th block of the haze-free image Get the dictionary of the i-th block of the foggy image The approximate distribution of (2).

4. The method according to claim 3, characterized in that The forward process of image fogging in step S200 includes: Step S211, obtaining the i-th block dictionary of the fog-free image and the dictionary of the i-th block of the foggy image The residual between ,Right now and The difference between the pixel values ​​​​between; Step S212: Establish a Markov chain with a length of T （3） Among them, t x ,t y are two different time steps, , when t x =T and t y =0, the input foggy image is defogged, and t x = 0 and t y =T, fog is added to the input fog-free image, and T is the maximum sampling step number; and is the corresponding disturbance data; when t x >1 hour, and , is a monotonically increasing sequence of hyperparameters; is the scaling factor, I is the identity matrix; Step S213, according to formula (3), by changing the adjustment parameter and hyperparameters Gradually increase the noise and the adjusted residuals , from the dictionary of the ith block of the foggy image Get the dictionary of the i-th block of the haze-free image The approximate distribution of （4）。 5. The method according to claim 4, characterized in that Step S300 includes: Step S301, setting optimization parameters Minimize the negative evidence lower bound of （6） in, Represents KL divergence calculation; target distribution and According to formula (5), the processing is explicitly expressed as （7） According to Bayes' theorem （8） Step S302: Obtain the predicted variance of the noise （9） Step S303, obtaining the mean value of noise prediction （10） in, is a of deep neural networks.

6. The method according to claim 5, characterized in that According to formula (10), the objective function of formula (6) is simplified to （11） in, is the weight.

7. The method according to claim 6, characterized in that 。

Images