A dehazing method based on stochastic differential equations and Gaussian pyramid

Abstract

The present invention relates to a defogging method based on the combination of stochastic differential equations and Gaussian pyramids. High-resolution and low-quality foggy images are collected and input into a Gaussian pyramid network for decomposition to generate a first multi-layer feature map and a second multi-layer feature map. The first multi-layer feature map is subjected to a diffusion process to generate a noise feature map. The second multi-layer feature map and the noise feature map are input into a NAFNet network for training to predict noise. The noise feature map is input into the NAFNet network and, through iterative processing, an initial state noise-free feature map is generated. The initial state noise-free feature map is input into a Gaussian pyramid network and, through layer-by-layer reconstruction, a defogging image is obtained. The present invention relates to a defogging method based on the combination of stochastic differential equations and Gaussian pyramids. High-resolution and low-quality foggy images are input into the Gaussian pyramid, and Fourier transform is introduced into the noise prediction network NAFNet. A loss function is introduced to optimize model parameters to improve the defogging effect of the image.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method for generating clear road traffic images in foggy road traffic scenes. Background Art

[0002] Haze reduces visibility and degrades image quality, severely impacting the performance of autonomous driving and surveillance systems. Traditional dehazing algorithms rely primarily on models and image processing techniques, modeling the scattering and attenuation of light in images to restore image clarity in hazy conditions. However, these algorithms rely on specific prior assumptions, resulting in unstable image dehazing results and artifacts and color distortion.

[0003] In recent years, diffusion models have demonstrated strong capabilities in image restoration and generation tasks. Through a step-by-step denoising process, diffusion models can generate clear, detailed, and high-quality images. However, diffusion models are slow to generate images, and the step-by-step denoising process requires multiple iterations. This places high demands on model size and computing resources, making training time-consuming and resource-intensive. Summary of the Invention

[0004] The technical problem of the present invention is to propose a defogging method to solve the problems of high computing resource consumption, slow generation speed, sensitivity to noise, unstable optimization, and slow training and inference speed of the diffusion model when processing high-resolution images.

[0005] The technical solution of the present invention is a defogging method based on a combination of stochastic differential equations and Gaussian pyramids, comprising the following steps:

[0006] S1: collects high-resolution and low-quality foggy images to form dataset D1;

[0007] S2: Input the data set D1 into the Gaussian pyramid network for decomposition to generate the first multi-layer feature map and the second multi-layer feature map;

[0008] S3: Perform a diffusion process on the first multi-layer feature map to generate a noise feature map;

[0009] S4: Input the second multi-layer feature map and the noise feature map into the NAFNet network for training, and then predict the noise;

[0010] S5: Input the noise feature map of step S3 into the NAFNet network, and generate the initial state noise-free feature map through iterative processing;

[0011] S6: Input the initial state noise-free feature map into the Gaussian pyramid network and reconstruct it layer by layer to obtain the reconstructed defogging image.

[0012] Preferably, in step S1, the high-resolution and low-quality foggy images in the dataset D1 have the same size, forming a one-to-one correspondence.

[0013] Furthermore, in step S2, the Gaussian pyramid network generates a set of images with gradually reduced resolution by performing Gaussian blur and downsampling operations on the original image, where the image resolution of each layer is half of the previous layer, thereby forming a pyramid-shaped multi-scale representation of the image; the high-resolution image is passed through the Gaussian pyramid network to generate a first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate a second multi-layer feature map.

[0014] Preferably, the diffusion process in step S3 is a forward diffusion process, in which the feature map with the lowest resolution is selected from the first multi-layer feature map, noise is gradually added according to the time step, and a noisy feature map with the time step is generated. The calculation formula of the forward diffusion process is:

[0015] ;

[0016] Where, represents the target clean feature map, x represents the feature map of the current time step, Indicates that the feature map deviates from the target, Indicates the speed at which the feature graph evolves, represents the noise intensity, Represents a random noise term used to simulate the randomness of noise.

[0017] Preferably, step S4 further includes a reverse diffusion process, inputting the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map into the NAFNet network to generate a noisy feature map, and then predicting the corresponding noise component from the noisy feature map, and gradually removing the noise to restore the clarity of the image. The reverse diffusion process is calculated as follows:

[0018] ;

[0019] Where, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of the data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

[0020] Preferably, the NAFNet network includes an encoder, an intermediate feature layer and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs a fast Fourier transform to obtain a first amplitude spectrum and a first phase spectrum; the intermediate feature layer processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs a fast Fourier transform again to obtain a second amplitude spectrum and a second phase spectrum, performs an inverse Fourier transform on the first phase spectrum and the second amplitude spectrum to generate a feature map, and inputs it into the decoder.

[0021] Preferably, step S4 further includes inputting the noise prediction result into a loss function for calculation to optimize the model parameters; the loss function includes:

[0022] 1) L1 loss: The L1 loss function is based on the average of the absolute differences between the predicted value and the true value, and is used to measure the difference between the reconstructed image and the original image;

[0023] 2) Lms-ssim loss: The Lms-ssim loss function is based on the structural similarity index. By combining the multi-scale structural similarity index information, it measures the similarity of images at multiple scales and is used to reconstruct images that are visually close to the original images.

[0024] 3) By combining L1 loss with Lms-ssim loss, the loss function optimizes the model performance to remove noise and improve the quality of image reconstruction.

[0025] Furthermore, step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a dehazed image.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1) This paper proposes a dehazing method based on a combination of stochastic differential equations and Gaussian pyramids. This method inputs high-resolution and low-quality foggy images into a Gaussian pyramid, eliminating the need for separate training. This method can process large images and addresses the issues of slow diffusion model generation and high computational resource consumption.

[0028] 2) This paper proposes a dehazing method based on the combination of stochastic differential equations and Gaussian pyramids. It introduces Fourier transform into the noise prediction network NAFNet to improve the model's ability to detect periodic structures and texture details, thereby increasing the effectiveness of image dehazing.

[0029] 3) This paper proposes a dehazing method based on the combination of stochastic differential equations and Gaussian pyramids, which introduces a loss function to optimize the model parameters, enabling the model to generate higher quality reconstructed images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below with reference to the accompanying drawings and examples.

[0031] Figure 1 The figure is a flow chart of a defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to an embodiment of the present invention.

[0032] Figure 2 This is a diagram showing the defogging effect of random images in the dataset cityscapes_foggy according to an embodiment of the present invention.

[0033] Figure 3 This is a diagram showing the dehazing effect of random images in the 4K-HAZE dataset according to an embodiment of the present invention.

[0034] Figure 4 This is a diagram showing the defogging effect of random images from the DKITTI dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] like Figure 1 As shown in FIG, a defogging method based on a combination of stochastic differential equations and Gaussian pyramids includes the following steps:

[0036] S1: Collect high-resolution images HR and low-quality foggy images LQ to form the dataset D1.

[0037] In step S1, the high-resolution image HR and the low-quality foggy image LQ in the dataset D1 have the same size, forming a one-to-one correspondence.

[0038] S2: The high-resolution clean image HR is input into the Gaussian Pyramid network Gaussian Pyramid1 for Gaussian pyramid decomposition to generate the first multi-layer feature map. The low-quality foggy image LQ is input into the Gaussian Pyramid network Gaussian Pyramid2 for Gaussian pyramid decomposition to generate the second multi-layer feature map.

[0039] In step S2, the Gaussian pyramid network generates a set of images with gradually reduced resolution by performing Gaussian blur and downsampling operations on the original image. The image resolution of each layer is half of the previous layer, thereby forming a pyramid-shaped multi-scale representation of the image; the high-resolution image is passed through the Gaussian pyramid network to generate the first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate the second multi-layer feature map.

[0040] S3: Perform a diffusion process on the first multi-layer feature map to generate a noise feature map.

[0041] The diffusion process in step S3 is a forward diffusion process. The feature map with the lowest resolution is selected from the first multi-layer feature map, and noise is gradually added according to the time step to generate a noisy feature map of the time step. The calculation formula of the forward diffusion process is:

[0042] ;

[0043] Where, Represents the target clean feature map, which remains fixed during the diffusion process and is the reference target of the feature map x at all time steps in the forward diffusion; x represents the feature map of the current time step. As the diffusion process proceeds, x is gradually contaminated by noise and deviates from the target feature map. , which eventually evolves into a completely noisy feature map; Represents the feature map The speed of evolution, that is, as time t changes, the feature map gradually deviates from the clean map; Indicates the noise intensity, which controls the amount of random noise added in each time step. Represents a random noise term used to simulate the randomness of noise.

[0044] The feature map Z0 with the lowest resolution in the first multi-layer feature map is diffused and noise is gradually added until the noise feature map Z at time T is generated. T .

[0045] S4: The feature map F with the lowest resolution in the second multi-layer feature map cond With Z T Input into the NAFNet network for training and noise prediction.

[0046] Step S4 also includes a reverse diffusion process, in which the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map are input into the NAFNet network to generate a noisy feature map, and then the corresponding noise component is predicted from the noisy feature map, and the noise is gradually removed to restore the clarity of the image. The reverse diffusion process is calculated as follows:

[0047] ;

[0048] Where, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of the data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

[0049] Step S4 also includes inputting the noise prediction result into the loss function Calculated in, used to optimize model parameters, the loss function includes:

[0050] 1) L1 loss: The L1 loss function is based on the average of the absolute differences between the predicted value and the true value, and is used to measure the difference between the reconstructed image and the original image;

[0051] 2) Lms-ssim loss: The Lms-ssim loss function is based on the structural similarity index. By combining multi-scale structural similarity index information, it measures the similarity of images at multiple scales and is used to reconstruct images that are visually close to the original images.

[0052] 3) By combining L1 loss with Lms-ssim loss, the loss function optimizes the model performance to remove noise and improve the quality of image reconstruction.

[0053] The NAFNet network includes an encoder Encoder-Layer4, an intermediate feature layer NAFBlock and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs a fast Fourier transform FFT to obtain a first amplitude spectrum Amp and a first phase spectrum Phase; in the intermediate feature layer, the encoder Encoder-Layer4 processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs a fast Fourier transform FFT again to obtain a second amplitude spectrum Amp1 and a second phase spectrum Phase1, and performs an inverse Fourier transform FFT on the first phase spectrum Phase and the second amplitude spectrum Amp1 to generate a feature map, which is input into the decoder.

[0054] S5: Input the noise feature map of step S3 into the NAFNet network to generate the feature map at time T-1 , and generate the initial state noise-free feature map through iterative processing .

[0055] S6: Initial state noise-free feature map Input PyramidReconstruction6 in the Gaussian pyramid network and reconstruct it layer by layer to obtain the reconstructed defogging image.

[0056] Step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a dehazed image.

[0057] To illustrate the effectiveness of the model constructed by the present invention, a comparative experiment is conducted to compare the image dehazing effects of the present invention with those of dark channel, AOD-Net, FFA-Net, EPDN, DehazeFormer, wavelet DehazeFormer, and Refuison. Images are randomly selected from the datasets cityscapes_foggy, 4K-HAZE, and DKITTI. The processing results are shown below:

[0058] like Figure 2 As shown in the figure, the dehazing effect of a random image from the dataset cityscapes_foggy is not thorough, and many details are obscured by the dark channel, AOD-Net, FFA-Net, DehazeFormer, and wavelet DehazeFormer methods. The EPDN and Refuison methods also fail to fully restore the upper part of the tree when generating a clean image. The method proposed in this paper effectively removes the haze, successfully retaining a large amount of detail, and the generated image is closer to a clear clean image. Table 1 shows the comparison of dehazing effects on the cityscapes_foggy dataset:

[0059]

[0060] like Figure 3 As shown in the dehazing effect diagram of random images in the dataset 4K-HAZE, the dark channel, AOD-Net, FFA-Net, DehazeFormer, and wavelet DehazeFormer dehazing algorithms failed to completely eliminate the haze, resulting in a large number of details being blocked, and the AOD-Net algorithm also had color deviation; the EPDN and Refuison methods had poor restoration effects in local areas when generating clean images, while the method proposed in the present invention did not have color deviation, and many details were generated, which was closer to a clean image.

[0061] The comparison data of dehazing effect in 4K-HAZE dataset is shown in Table 2:

[0062]

[0063] like Figure 4 As shown in the dehazing effect diagram of random images of the dataset DKITTI, the dark channel and AOD-Net dehazing algorithms failed to completely eliminate the haze, resulting in a large number of details being blocked, and FFA-Net, DehazeFormer, wavelet DehazeFormer, EPDN and Refuison showed mosaics and blurs in distant details. The method proposed in this invention performed well in processing distant details, retained rich details, and made the generated image closer to a clear and clean image.

[0064] The comparison data of dehazing effect in DKITTI dataset is shown in Table 3:

[0065]

[0066] The results show that the proposed method is significantly superior to the existing technology in terms of indicators and restored images, proving the feasibility and effectiveness of the algorithm.

[0067] The above embodiments are merely preferred technical solutions of the present invention and should not be construed as limiting the present invention. The scope of protection of the present invention shall be the technical solutions recited in the claims, including equivalent alternatives to the technical features of the technical solutions recited in the claims. In other words, equivalent alternatives and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A defogging method based on the combination of stochastic differential equations and Gaussian pyramids, characterized in that: The following steps are involved: S1: collects high-resolution and low-quality foggy images to form dataset D1; S2: Dataset D1 is fed into a Gaussian pyramid network. By performing Gaussian blurring and downsampling operations on the original image, a set of images with gradually lower resolutions is generated. The resolution of each layer is half of the previous layer, thus forming a pyramid-shaped multi-scale representation of the image. The high-resolution image is passed through the Gaussian pyramid network to generate the first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate the second multi-layer feature map. S3: Perform a diffusion process on the first multi-layer feature map to generate a noise feature map; S4: Input the second multi-layer feature map and the noise feature map into the NAFNet network for training, predict the noise, and input the prediction results into the loss function for calculation to optimize the model parameters; the NAFNet network includes an encoder, an intermediate feature layer and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs a fast Fourier transform to obtain a first amplitude spectrum and a first phase spectrum; the intermediate feature layer processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs a fast Fourier transform again to obtain a second amplitude spectrum and a second phase spectrum, performs an inverse Fourier transform on the first phase spectrum and the second amplitude spectrum to generate a feature map, and inputs it into the decoder; the loss function includes: L1 loss, which is based on the average of the absolute difference between the predicted value and the true value, and is used to measure the difference between the reconstructed image and the original image; L ms-ssim Loss, based on the structural similarity index, measures the similarity of images at multiple scales by combining multi-scale structural similarity index information, and is used to reconstruct images that are visually close to the original images; by combining L1 loss with L ms-ssim The loss function is combined to optimize the model performance, which is used to remove noise and improve the quality of image reconstruction; S5: Input the noise feature map of step S3 into the NAFNet network, and generate the initial state noise-free feature map through iterative processing; S6: Input the initial state noise-free feature map into the Gaussian pyramid network and reconstruct it layer by layer to obtain the reconstructed defogging image.

2. The defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to claim 1, characterized in that: In step S1, the high-resolution and low-quality foggy images in the dataset D1 have the same size, forming a one-to-one correspondence.

3. The defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to claim 1, characterized in that: In step S3, the diffusion process is a forward diffusion process, in which the feature map with the lowest resolution is selected from the first multi-layer feature map, noise is gradually added according to the time step, and a noisy feature map with the time step is generated. The calculation formula of the diffusion process is: The calculation formula for the forward diffusion process is: ； Where, represents the target clean feature map, x represents the feature map of the current time step, Indicates that the feature map deviates from the target, Indicates the speed at which the feature graph evolves, represents the noise intensity, Represents a random noise term used to simulate the randomness of noise.

4. The defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to claim 1, characterized in that: The step S4 also includes a reverse diffusion process, which inputs the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map into the NAFNet network to generate a noisy feature map, and then predicts the corresponding noise component from the noisy feature map, and gradually removes the noise to restore the clarity of the image. The reverse diffusion process is calculated as follows: ； Where, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of the data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

5. The defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to claim 1, characterized in that: The step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a defogged image.

Images