Defogging method based on combination of stochastic differential equation and Gaussian pyramid

Abstract

The invention relates to a defogging method based on the combination of a stochastic differential equation and a Gaussian pyramid, and the method comprises the steps: collecting a high-resolution and low-quality foggy picture, inputting the foggy picture into a Gaussian pyramid network for decomposition, and generating a first multi-layer feature map and a second multi-layer feature map; performing a diffusion process on the first multi-layer feature map to generate a noise feature map; inputting the second multi-layer feature map and the noise feature map into an NAFNet network for training so as to predict noise; inputting the noise feature map into an NAFNet network, and generating an initial state noise-free feature map through iterative processing; and inputting the initial state noiseless feature map into a Gaussian pyramid network, and performing layer-by-layer reconstruction to obtain a defogged picture. According to the defogging method based on the combination of the stochastic differential equation and the Gaussian pyramid, the high-resolution and low-quality foggy picture is input into the Gaussian pyramid, Fourier transform is introduced into the noise prediction network NAFNet, a loss function is introduced to optimize model parameters, and the defogging effect of the picture is improved.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a method for generating clear road traffic images in foggy road traffic scenes. Background Art

[0002] Haze weather causes reduced visibility and deteriorated image quality, which seriously affects the performance of autonomous driving and monitoring. Traditional dehazing algorithms are mainly based on models and image processing technology, which restore image clarity in haze weather by modeling the scattering and attenuation process of light in images. However, such algorithms rely on specific prior assumptions, resulting in unstable image dehazing effects, artifacts, and color distortion.

[0003] In recent years, diffusion models have shown strong capabilities in image restoration and generation tasks. Through the step-by-step denoising process, diffusion models can generate clear, delicate and high-quality images. However, the generation speed of diffusion models is slow, the step-by-step denoising process requires multiple iterative calculations, and the requirements for model size and computing resources are high. The training process is time-consuming and resource-intensive. Summary of the invention

[0004] The technical problem of the present invention is to propose a defogging method to solve the problems of high computing resource consumption, slow generation speed, sensitivity to noise, unstable optimization, and slow training and inference speed when processing high-resolution images in a diffusion model.

[0005] The technical solution of the present invention is a defogging method based on the combination of stochastic differential equations and Gaussian pyramids, comprising the following steps: S1: collects high-resolution and low-quality foggy images to form dataset D1; S2: Input the data set D1 into the Gaussian pyramid network for decomposition to generate the first multi-layer feature map and the second multi-layer feature map; S3: performing a diffusion process on the first multi-layer feature map to generate a noise feature map; S4: Input the second multi-layer feature map and the noise feature map into the NAFNet network for training, and then predict the noise; S5: Input the noise feature map of step S3 into the NAFNet network, and generate an initial state noise-free feature map through iterative processing; S6: Input the initial state noise-free feature map into the Gaussian pyramid network, and reconstruct it layer by layer to obtain the reconstructed dehazed image.

[0006] Preferably, in step S1, the high-resolution and low-quality foggy images in the data set D1 have the same size, forming a one-to-one correspondence.

[0007] Furthermore, in step S2, the Gaussian pyramid network generates a set of images with gradually reduced resolution by performing Gaussian blur and downsampling operations on the original image, wherein the image resolution of each layer is half of the previous layer, thereby forming a pyramid-shaped multi-scale representation of the image; the high-resolution image is passed through the Gaussian pyramid network to generate the first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate the second multi-layer feature map.

[0008] Preferably, the diffusion process in step S3 is a forward diffusion process, in which the feature map with the lowest resolution is selected from the first multi-layer feature map, and noise is gradually added according to the time step to generate a noisy feature map with the time step. The calculation formula of the forward diffusion process is: ; In the formula, represents the target clean feature map, x represents the feature map of the current time step, Indicates that the feature map deviates from the target, Indicates the speed at which the feature graph evolves. represents the noise intensity, Represents a random noise term, which is used to simulate the randomness of noise.

[0009] Preferably, step S4 further includes a reverse diffusion process, in which the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map are input into the NAFNet network to generate a noisy feature map, and then the corresponding noise component is predicted from the noisy feature map, and the noise is gradually removed to restore the clarity of the image. The reverse diffusion process is calculated as follows: ; In the formula, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

[0010] Preferably, the NAFNet network includes an encoder, an intermediate feature layer and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs a fast Fourier transform to obtain a first amplitude spectrum and a first phase spectrum; the intermediate feature layer processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs a fast Fourier transform again to obtain a second amplitude spectrum and a second phase spectrum, performs an inverse Fourier transform on the first phase spectrum and the second amplitude spectrum to generate a feature map, and inputs it into the decoder.

[0011] Preferably, step S4 further includes inputting the prediction result of the noise into a loss function for calculation to optimize the model parameters; the loss function includes: 1) L1 loss: The L1 loss function is based on the average of the absolute difference between the predicted value and the true value, which is used to measure the difference between the reconstructed image and the original image; 2) Lms-ssim loss: The Lms-ssim loss function is based on the structural similarity index. It measures the similarity of images at multiple scales by combining multi-scale structural similarity index information, and is used to reconstruct images that are visually close to the original images.

[0012] 3) By combining L1 loss with Lms-ssim loss, the loss function optimizes the model performance for removing noise and improving the quality of image reconstruction.

[0013] Furthermore, step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a dehazed image.

[0014] Compared with the prior art, the beneficial effects of the present invention include: 1) The present invention proposes a defogging method based on the combination of stochastic differential equations and Gaussian pyramids, which inputs high-resolution and low-quality foggy images into Gaussian pyramids. It does not require separate training and can process large-size images, solving the problems of slow diffusion model generation and high computing resource consumption.

[0015] 2) The present invention proposes a defogging method based on the combination of stochastic differential equations and Gaussian pyramids, which introduces Fourier transform into the noise prediction network NAFNet to improve the model's ability to capture periodic structures and texture details, thereby increasing the effectiveness of image defogging.

[0016] 3) The present invention proposes a dehazing method based on the combination of stochastic differential equations and Gaussian pyramids, which introduces a loss function to optimize the model parameters and enables the model to generate higher quality reconstructed images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0018] Figure 1 The figure is a flow chart of a defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to an embodiment of the present invention.

[0019] Figure 2 This is a diagram showing the defogging effect of random images in the dataset cityscapes_foggy according to an embodiment of the present invention.

[0020] Figure 3 This is a diagram showing the dehazing effect of random images of the 4K-HAZE dataset according to an embodiment of the present invention.

[0021] Figure 4 This is a diagram showing the defogging effect of a random image of the DKITTI dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, a defogging method based on the combination of stochastic differential equations and Gaussian pyramids includes the following steps: S1: Collect high-resolution images HR and low-quality foggy images LQ to form the dataset D1.

[0023] In step S1, the high-resolution image HR and the low-quality foggy image LQ of the data set D1 have the same size, forming a one-to-one correspondence.

[0024] S2: The high-resolution clean image HR is input into the Gaussian pyramid network Gaussian Pyramid1 for Gaussian pyramid decomposition to generate the first multi-layer feature map, and the low-quality foggy image LQ is input into the Gaussian pyramid network GaussianPyramid2 for Gaussian pyramid decomposition to generate the second multi-layer feature map.

[0025] In step S2, the Gaussian pyramid network generates a set of images with gradually reduced resolution by performing Gaussian blur and downsampling operations on the original image, where the image resolution of each layer is half of the previous layer, thereby forming a pyramid-shaped multi-scale representation of the image; the high-resolution image is passed through the Gaussian pyramid network to generate the first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate the second multi-layer feature map.

[0026] S3: Perform a diffusion process on the first multi-layer feature map to generate a noise feature map.

[0027] The diffusion process in step S3 is a forward diffusion process. The feature map with the lowest resolution is selected from the first multi-layer feature map, and noise is gradually added according to the time step to generate a noisy feature map with the time step. The calculation formula of the forward diffusion process is: ; In the formula, represents the target clean feature map, which remains fixed during the diffusion process and is the reference target of the feature map x at all time steps in the forward diffusion; x represents the feature map of the current time step. As the diffusion process proceeds, x is gradually contaminated by noise and deviates from the target feature map. , which eventually evolves into a completely noisy feature map; Represents the feature map The speed of evolution, that is, as time t changes, the feature map gradually deviates from the clean map; Represents the noise intensity, which controls the amount of random noise added at each time step. Represents a random noise term, which is used to simulate the randomness of noise.

[0028] The feature map Z0 with the lowest resolution in the first multi-layer feature map is diffused and noise is gradually added until the noise feature map Z at time T is generated. T .

[0029] S4: The feature map F with the lowest resolution in the second multi-layer feature map cond With Z T Input into the NAFNet network for training and noise prediction.

[0030] Step S4 also includes a reverse diffusion process, in which the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map are input into the NAFNet network to generate a noisy feature map, and then the corresponding noise component is predicted from the noisy feature map, and the noise is gradually removed to restore the clarity of the image. The reverse diffusion process is calculated as follows: ; In the formula, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

[0031] Step S4 also includes inputting the noise prediction result into the loss function Calculated in, used to optimize model parameters, the loss function includes: 1) L1 loss: The L1 loss function is based on the average of the absolute difference between the predicted value and the true value, which is used to measure the difference between the reconstructed image and the original image; 2) Lms-ssim loss: The Lms-ssim loss function is based on the structural similarity index. It measures the similarity of images at multiple scales by combining multi-scale structural similarity index information, and is used to reconstruct images that are visually close to the original images. 3) By combining L1 loss with Lms-ssim loss, the loss function optimizes the model performance for removing noise and improving the quality of image reconstruction.

[0032] The NAFNet network includes an encoder Encoder-Layer4, an intermediate feature layer NAFBlock and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs fast Fourier transform FFT to obtain the first amplitude spectrum Amp and the first phase spectrum Phase; in the intermediate feature layer, the encoder Encoder-Layer4 processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs fast Fourier transform FFT again to obtain the second amplitude spectrum Amp1 and the second phase spectrum Phase1, and performs inverse Fourier transform FFT on the first phase spectrum Phase and the second amplitude spectrum Amp1 to generate a feature map, and input it into the decoder.

[0033] S5: Input the noise feature map of step S3 into the NAFNet network to generate the feature map at time T-1 , and generate the initial state noise-free feature map through iterative processing .

[0034] S6: Initial state noise-free feature map Input PyramidReconstruction6 in the Gaussian pyramid network and reconstruct it layer by layer to obtain the reconstructed dehazed image.

[0035] Step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid Pyramid Reconstruction layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a dehazed image.

[0036] In order to illustrate the effect of the model constructed by the present invention, the comparative experiment compares the image dehazing effect of the present invention with that of dark channel, AOD-Net, FFA-Net, EPDN, DehazeFormer, wavelet DehazeFormer and Refuison, and randomly selects pictures from the datasets cityscapes_foggy, 4K-HAZE and DKITTI. The processing effects are shown as follows: like Figure 2As shown in the figure, the dehazing effect of random images in the dataset cityscapes_foggy is not thorough in dehazing by dark channel, AOD-Net, FFA-Net, DehazeFormer, and wavelet DehazeFormer, and a large number of details are blocked; when the EPDN and Refuison methods generate clean images, the content above the tree in the image is not fully restored. The method proposed in the present invention effectively removes the haze, successfully retains a large number of details, and the generated image is closer to a clear and clean image. The comparison data of the dehazing effect in the dataset cityscapes_foggy is shown in Table 1:

[0037] like Figure 3 As shown in the figure, the dehazing effect diagram of random images of the data set 4K-HAZE shows that the dark channel, AOD-Net, FFA-Net, DehazeFormer, and wavelet DehazeFormer dehazing algorithms failed to completely eliminate the haze, resulting in a large number of details being blocked, and the AOD-Net algorithm also had color deviation; when the EPDN and Refuison methods generated clean images, the restoration effect of local areas was poor. The method proposed in the present invention did not have color deviation, and many details were generated, which was closer to a clean image.

[0038] The comparison data of dehazing effect in 4K-HAZE dataset is shown in Table 2:

[0039] like Figure 4 As shown in the figure, the dehazing effect of random images of the dataset DKITTI, the dark channel and AOD-Net dehazing algorithms failed to completely eliminate the haze, resulting in a large number of details being blocked, and FFA-Net, DehazeFormer, wavelet DehazeFormer, EPDN and Refuison showed mosaic and blur in the distant details. The method proposed in the present invention performs well in processing distant details, retains rich details, and makes the generated image closer to a clear and clean image.

[0040] The comparison data of dehazing effect in DKITTI dataset is shown in Table 3:

[0041] The results show that the proposed method is significantly superior to the existing technology in terms of indicators and restored images, proving the feasibility and effectiveness of the algorithm.

[0042] The above embodiments are only preferred technical solutions of the present invention and should not be regarded as limitations of the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A defogging method based on the combination of stochastic differential equations and Gaussian pyramids, characterized in that: The following steps are involved: S1: collects high-resolution and low-quality foggy images to form dataset D1; S2: Input the data set D1 into the Gaussian pyramid network for decomposition to generate the first multi-layer feature map and the second multi-layer feature map; S3: performing a diffusion process on the first multi-layer feature map to generate a noise feature map; S4: Input the second multi-layer feature map and the noise feature map into the NAFNet network for training, and then predict the noise; S5: Input the noise feature map of step S3 into the NAFNet network, and generate an initial state noise-free feature map through iterative processing; S6: Input the initial state noise-free feature map into the Gaussian pyramid network, and reconstruct it layer by layer to obtain the reconstructed dehazed image.

2. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 1, characterized in that: In step S1, the high-resolution and low-quality foggy images in the data set D1 have the same size, forming a one-to-one correspondence.

3. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 1, characterized in that: In step S2, the Gaussian pyramid network generates a set of images with reduced resolution layer by layer by performing Gaussian blur and downsampling operations on the original image, and the image resolution of each layer is half of the previous layer, thereby forming a pyramid-shaped multi-scale representation of the image; the high-resolution image is passed through the Gaussian pyramid network to generate a first multi-layer feature map, and the low-quality foggy image is passed through the Gaussian pyramid network to generate a second multi-layer feature map.

4. The defogging method based on the combination of stochastic differential equations and Gaussian pyramids according to claim 1, characterized in that: In step S3, the diffusion process is a forward diffusion process, in which the feature map with the lowest resolution is selected from the first multi-layer feature map, noise is gradually added according to the time step, and a noisy feature map with the time step is generated. The calculation formula of the diffusion process is: The calculation formula for the forward diffusion process is: ； In the formula, represents the target clean feature map, x represents the feature map of the current time step, Indicates that the feature map deviates from the target, Indicates the speed at which the feature graph evolves. represents the noise intensity, Represents a random noise term, which is used to simulate the randomness of noise.

5. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 1, characterized in that: The step S4 also includes a reverse diffusion process, in which the feature map with the lowest resolution and the noise feature map in the second multi-layer feature map are input into the NAFNet network to generate a noisy feature map, and then the corresponding noise component is predicted from the noisy feature map, and the noise is gradually removed to restore the clarity of the image. The reverse diffusion process is calculated as follows: ； In the formula, represents the target clean feature map, x represents the current noisy feature map, represents the recovery speed of the eigenvector, Indicates the rate at which the feature map approaches the clean feature map, represents the noise intensity, represents the ladder of the data distribution, represents the change of image features at time t, represents a random noise term that models the residual randomness.

6. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 5, characterized in that: The NAFNet network includes an encoder, an intermediate feature layer and a decoding part; the encoder outputs the feature map with the lowest resolution through the encoder, and performs a fast Fourier transform to obtain a first amplitude spectrum and a first phase spectrum; the intermediate feature layer processes the feature map with the lowest resolution through the intermediate feature layer NAFBlock, and performs a fast Fourier transform again to obtain a second amplitude spectrum and a second phase spectrum, performs an inverse Fourier transform on the first phase spectrum and the second amplitude spectrum to generate a feature map, and inputs it into the decoder.

7. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 6, characterized in that: The step S4 further includes inputting the noise prediction result into a loss function for calculation to optimize the model parameters. The loss function includes: 1) L1 loss: The L1 loss function is based on the average of the absolute difference between the predicted value and the true value, which is used to measure the difference between the reconstructed image and the original image; 2) L ms-ssim Loss: L ms-ssim The loss function is based on the structural similarity index. It measures the similarity of images at multiple scales by combining the multi-scale structural similarity index information, and is used to reconstruct images that are visually close to the original images. 3) By combining L1 loss with L ms-ssim The loss function optimizes the model performance and is used to remove noise and improve the quality of image reconstruction.

8. The defogging method based on the combination of stochastic differential equation and Gaussian pyramid according to claim 1, characterized in that: The step S6 includes upsampling the lowest resolution feature map obtained by decomposing the Gaussian pyramid layer by layer, fusing it with the adjacent high-resolution feature map, and finally generating a dehazed image.

Images