An image defogging method based on laplace image decomposition and diffusion model

Abstract

The application discloses an image defogging method based on a Laplace image decomposition and diffusion model, and belongs to the technical field of image processing, and comprises the following steps: S1, data preprocessing; S2, Laplace image decomposition; S3, construction of an image defogging network; and S4, image defogging.The method decomposes a foggy image into a low-pass image and a high-frequency residual part, reasonably designs different modules for recovery, and reduces the difficulty of defogging; the powerful generation capacity of the diffusion model is used to only act on the low-pass image, and the inference speed of the diffusion model is significantly accelerated; and the advantage of the Laplace image pyramid in preserving original information is used, so that the reconstructed clean image is more faithful, and the method has a wide application prospect in the field of image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image dehazing method based on Laplacian image decomposition and diffusion model. Background Technology

[0002] With the continuous development of computer vision and image processing technologies, image dehazing has become an important research direction in many practical applications. Haze can lead to reduced image contrast and color distortion, affecting the visual quality of images and subsequent automated processing, such as target recognition, scene understanding, and navigation systems. Traditional image dehazing methods rely on statistically discovered prior knowledge to predict transmission maps and global atmospheric light, and finally combine atmospheric scattering models to recover haze-free images. However, this prior knowledge does not always fit various haze scenarios, easily causing problems such as color distortion. In recent years, the development of deep learning technology has provided new ideas for image dehazing. Researchers have begun to design complex convolutional neural networks (CNNs) or Transformer networks to directly recover clean images in an end-to-end manner. However, these methods usually have limited performance when dealing with complex haze scenarios, especially in high-concentration haze or non-uniformly distributed haze environments, where the dehazing effect is difficult to achieve the expected results.

[0003] A search revealed that Chinese invention patent CN109712083B discloses a single-image dehazing method based on a convolutional neural network. Its network design includes a shallow neural network model and a deep neural network model. The shallow network model extracts and fuses features of the color space of the hazy image to obtain a scene depth map of the hazy image. The deep network model performs multi-scale mapping, pooling, and convolution operations on the scene depth map, outputting a transmittance map of the hazy image. Finally, a haze-free image is recovered based on an atmospheric scattering model. However, the distribution of haze in real-world scenes is often more complex, deviating from the ideal data assumptions of the atmospheric scattering model, resulting in poor dehazing performance.

[0004] Chinese invention patent CN113066028B discloses an image dehazing method based on Transformer deep neural network. It utilizes the inherent global feature extraction capability of Transformer and assists network training by reconstructing hazy images. However, Transformer network requires larger and higher quality data for training. Paired hazy / foggy images in real scenes are difficult to obtain, the training set is small, which limits the performance of the network model and produces serious color deviation.

[0005] Diffusion models have gradually attracted researchers' attention due to their superior performance in high-quality image generation and denoising. By simulating the gradual evolution of data, diffusion models can progressively recover detailed information in images, showing significant advantages, especially in handling noise or degradation problems. Compared with other deep learning models, diffusion models exhibit stronger adaptability and robustness, effectively addressing the shortcomings of traditional dehazing methods under complex haze conditions. However, traditional diffusion models suffer from slow inference speeds, requiring substantial computational resources and time overhead.

[0006] Based on the above analysis, for the more complex haze distribution in real-world scenarios, existing technologies require an image dehazing method that is more adaptable and computationally efficient. Therefore, this paper proposes an image dehazing method based on Laplacian image decomposition and a diffusion model. Summary of the Invention

[0007] The technical problem to be solved by this invention is: how to overcome the limitations of traditional deep neural networks in terms of limited feature extraction capability and slow inference speed of diffusion models in complex haze scenes, and thus solve problems such as insufficient dehazing and color distortion, and provide an image dehazing method based on Laplacian image decomposition and diffusion model.

[0008] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0009] S1: Data Preprocessing

[0010] Create a training dataset and preprocess the images;

[0011] S2: Laplacian Image Decomposition

[0012] The training images are decomposed using Laplacian image decomposition to obtain low-pass images and high-frequency residuals.

[0013] S3: Constructing an image dehazing network

[0014] An image dehazing network is constructed and trained to obtain an image dehazing model. The image dehazing network includes a diffusion model dehazing network, a high-frequency residual recovery network, and a low-frequency refinement network. The low-pass image and the high-frequency residual are respectively fed into the diffusion model dehazing network and the high-frequency residual recovery network for reconstruction to obtain the restored low-pass image and high-frequency residual. Starting from the low-pass image, the inverse transformation of Laplacian image decomposition is performed to obtain the preliminary restored image. Finally, the low-frequency refinement network is used to process the preliminary restored image to obtain the final dehazed image.

[0015] S4: Image Dehazing

[0016] Input the foggy image to be dehazed into the image dehazing model to obtain the dehazed image.

[0017] Furthermore, in step S1, the preprocessing process involves randomly cropping the image to 512×512 and performing data augmentation by horizontal flipping and random rotation.

[0018] Furthermore, in step S2, the original image is downsampled to generate a series of images with progressively lower resolution, which form a Gaussian pyramid. For adjacent layers in the Gaussian pyramid, the following operation is performed: the next layer is upsampled to the same size as the current layer, and then the upsampled result is subtracted from the current layer. The specific process is defined as follows:

[0019] G i+1 =Down(G i );

[0020] L i =G i -Up(G i+1 );

[0021] Where Down() and Up() represent downsampling and upsampling operations, respectively, G i For the downsampled low-pass image, L i L represents the high-frequency information lost during downsampling, i.e., the high-frequency residual. i This forms the Laplace image pyramid.

[0022] Furthermore, in step S3, the diffusion model dehazing network adopts a U-Net structure design, which operates only on the low-pass image to achieve overall haze removal, including a forward noise addition process and a reverse noise reduction process. The specific execution process is as follows:

[0023] In the paired training images, the low-pass image obtained by decomposing the haze-free image is denoised, and then the low-pass image of the haze image is used as a condition. The two are concatenated in the channel dimension to form a 6-channel tensor as the input of the diffusion model dehazing network. Guided by the haze features, the diffusion model dehazing network can reconstruct a clean low-pass image after iterative denoising.

[0024] Furthermore, the noise-adding process is defined as follows:

[0025]

[0026]

[0027] Where X0 is the low-pass image obtained from the decomposition of the haze-free image, X t This is the image after adding Gaussian noise to X0, where t is the time step and I is the identity matrix. Represents a Gaussian distribution, β t It is a predefined variance scheduling, α t =1-β t , This indicates that a noise ∈ is randomly sampled from a Gaussian distribution. t ;

[0028] During the denoising process, the diffusion model dehazing network iteratively predicts the noise intensity at each time step, ultimately obtaining a clean output. In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean. The prediction method for the mean is defined as follows:

[0029]

[0030] in, The low-pass image representing a foggy image, ∈ θ This represents the noise in the parameterized network predictions.

[0031] Furthermore, in step S3, the high-frequency residual recovery network adopts a U-Net structure design. First, it uses standard convolution to extract shallow features, then performs three downsampling and upsampling operations. After each upsampling and downsampling, residual dense blocks are used for feature enhancement. Finally, standard convolution is used to recover the output into a 3-channel output.

[0032] Furthermore, in step S3, when training the diffusion model dehazing network and the high-frequency residual recovery network, the following combined loss function is used:

[0033] L total =L noise +L low +L high +L image

[0034] Among them, L noise For noise prediction loss, L low For low-pass image reconstruction loss, Acting on the diffusion model dehazing network; L high For high-frequency reconstruction loss, For the high-frequency residuals predicted by the network, Its corresponding true value is applied to the high-frequency residual recovery network; L image For image reconstruction loss, I0 represents a clean, fog-free image. The image is a preliminary restored image after being processed by the diffusion model dehazing network and the high-frequency residual recovery network. The data is processed by both the diffusion model dehazing network and the high-frequency residual recovery network.

[0035] When training a low-frequency refined network, the following loss function is used:

[0036]

[0037] in, This is the final dehazed image.

[0038] Furthermore, in step S3, the specific processing procedure of the low-frequency refinement network for the initially restored image is as follows: Haar wavelet transform is performed on the obtained initially restored image to extract low-frequency information A. low Then, standard convolution is used to extract shallow features, followed by multi-scale feature extraction through three parallel residual blocks with dilated convolutions. The obtained features... It is obtained by splicing along the channel dimension. Finally, after two standard convolutions to reduce the number of channels, the enhanced low-frequency information is reconstructed. The final dehazed image can be obtained by performing an inverse wavelet transform. The entire processing procedure is defined as follows:

[0039]

[0040] F A =Conv(A low );

[0041]

[0042]

[0043]

[0044]

[0045] Where DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform, respectively; Conv() represents standard convolution; Dilated() represents dilated convolution; the dilation rates of the dilated convolutions in the three parallel residual blocks with dilated convolutions are 1, 2, and 3, respectively; and Concat() represents concatenation along the channel dimension. This is the final dehazed image.

[0046] The present invention has the following advantages over the prior art:

[0047] (1) This invention reduces the difficulty of dehazing by decomposing a foggy image into a low-pass image and a high-frequency residual part, and rationally designing different modules to restore them separately.

[0048] (2) This invention utilizes the powerful generation capability of the diffusion model and applies it only to low-pass images, which significantly accelerates the inference speed of the diffusion model.

[0049] (3) This invention utilizes the advantage of the Laplacian image pyramid in preserving the original information, making the reconstructed clean image more faithful and having broad application prospects in image processing and other fields. Attached Figure Description

[0050] Figure 1 This is a schematic flowchart of an image dehazing method based on Laplacian image decomposition and diffusion model according to the present invention;

[0051] Figure 2 This is a schematic diagram of the Laplacian image decomposition process in this invention;

[0052] Figure 3 This is a schematic diagram of the image dehazing network constructed in this invention, wherein the image dehazing network includes a diffusion model dehazing network, a high-frequency residual recovery network, and a low-frequency refinement network;

[0053] Figure 4 This is a schematic diagram of the high-frequency residual recovery network in this invention;

[0054] Figure 5 This is a schematic diagram of the low-frequency refining network in this invention. Detailed Implementation

[0055] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0056] like Figure 1 As shown, this embodiment of an image dehazing method based on Laplacian image decomposition and diffusion model specifically includes the following steps:

[0057] Step 1: Create a training dataset and preprocess the images: First, randomly crop the images to 512×512 size, and perform data augmentation by horizontal flipping and random rotation.

[0058] Step 2: Perform Laplacian image decomposition on the training images to obtain a low-resolution low-pass image and high-frequency residual components. The specific implementation is as follows: Figure 2 As shown:

[0059] In this embodiment, the scale of Laplacian image decomposition is 2. First, the original image is downsampled to generate a series of images with progressively decreasing resolution, forming a Gaussian pyramid. For adjacent layers in the Gaussian pyramid, the following operation is performed: the next layer is upsampled to the same size as the current layer, and then the upsampled result is subtracted from the current layer. This process preserves the high-frequency details lost during downsampling, ensuring no information loss during decomposition. The specific process can be defined as follows:

[0060] G i+1 =Down(G i );

[0061] L i =G i -Up(G i+1 );

[0062] Where Down() and Up() represent downsampling and upsampling operations, respectively, G i For the downsampled low-pass image, L i L represents the high-frequency information lost during downsampling, i.e., the high-frequency residual. i This forms the Laplace image pyramid.

[0063] Step 3: Construct a diffusion model dehazing network and a high-frequency residual recovery network. Feed the low-pass image and high-frequency residual components into the diffusion model dehazing network and the high-frequency residual recovery network respectively for reconstruction. The entire network execution process is as follows: Figure 3 As shown, the specific structure of the high-frequency residual recovery network is as follows: Figure 4 As shown.

[0064] In this embodiment, the specific processing procedure of step 3 is as follows:

[0065] Step 3-1: The constructed diffusion model dehazing network only operates on the low-pass image to achieve overall haze removal, including a forward noise addition process and a reverse noise reduction process. The specific execution process is as follows:

[0066] In the paired training images, the low-pass image obtained from the decomposition of the haze-free image is denoised, and then the low-pass image of the haze image is used as a condition. The two images are concatenated along the channel dimension to form a 6-channel tensor, which serves as the input to the diffusion model dehazing network. Guided by the haze features, the diffusion model dehazing network can reconstruct a clean low-pass image after iterative denoising. The denoising process is defined as follows:

[0067]

[0068]

[0069] Where X0 is the low-pass image obtained from the decomposition of the haze-free image, X t This is the image after adding Gaussian noise to X0, where t is the time step and I is the identity matrix. Represents a Gaussian distribution, β t It is a predefined variance scheduling, α t =1-β t , This indicates that a noise ∈ is randomly sampled from a Gaussian distribution. t ;

[0070] During the denoising process, the diffusion model dehazing network iteratively predicts the noise intensity at each time step, ultimately obtaining a clean output. In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean. The prediction of the mean can be defined as:

[0071]

[0072] in, The low-pass image representing a foggy image, ∈ θ This represents the noise in the parameterized network predictions.

[0073] Step 3-2: The high-frequency residual recovery network adopts the U-Net structure design. First, it uses standard convolution to extract shallow features, then performs three downsampling and upsampling operations. After each upsampling and downsampling, residual dense blocks (RDB) are used for feature enhancement. Finally, standard convolution is used to restore the output to 3 channels.

[0074] Step 3-3: Based on the restored low-pass image and high-frequency residual, starting from the low-pass image, the preliminary restored image can be obtained by performing an inverse Laplacian image decomposition transformation.

[0075] Step 4: Train the network model built in Step 3 using the PyTorch deep learning framework. The Adam optimizer is selected. For the diffusion model dehazing network operating on low-pass images, in addition to using the noise prediction loss, a low-pass image reconstruction loss is also used for constraint, which can be defined as:

[0076]

[0077]

[0078] in, This is a low-pass image predicted by sampling.

[0079] For high-frequency residual recovery networks, using high-frequency reconstruction loss as a constraint, it can be defined as:

[0080]

[0081] in, For the high-frequency residuals predicted by the network, For its corresponding true value, k is 2 in this embodiment. Furthermore, the image reconstruction loss is used to constrain the two networks, defined as:

[0082]

[0083] Where I0 represents a clean, fog-free image. This is the preliminary restored image after processing by the diffusion model dehazing network and the high-frequency residual recovery network. The total loss function is the sum of the above loss functions, defined as:

[0084] L total =L noise +L low +L high +L image .

[0085] Step 5: Construct a low-frequency refinement network and apply it to the preliminary restored image obtained in Step 3. For example... Figure 5 As shown, the low-frequency information A is first extracted from the preliminary restored image obtained in step 3 by performing Haar wavelet transform. low Then, standard convolution is used to extract shallow features, followed by multi-scale feature extraction through three parallel residual blocks with dilated convolutions. The obtained features... It is obtained by splicing along the channel dimension. Finally, after two standard convolutions to reduce the number of channels, the enhanced low-frequency information is reconstructed. The final dehazed image can be obtained by performing an inverse wavelet transform. The entire process is defined as follows:

[0086]

[0087] F A =Conv(A low );

[0088]

[0089]

[0090]

[0091]

[0092] In this context, DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform, respectively; Conv() represents standard convolution; Dilated() represents dilated convolution; the dilation rates of dilated convolutions in Branch1, Branch2, and Branch3 are 1, 2, and 3, respectively; and Concat() represents concatenation along the channel dimension. This is the final dehazed image.

[0093] Step 6: Train the low-frequency refined network to obtain the weight parameters. The training environment settings are the same as in Step 4, and the loss function is defined as:

[0094]

[0095] Step 7: Take a foggy image as input to the model and reconstruct a high-fidelity fog-free image as output using the weight parameters learned in Steps 4 and 6.

[0096] In summary, the image dehazing method based on Laplacian image decomposition and diffusion model described in the above embodiments reduces the difficulty of dehazing by decomposing a hazy image into a low-pass image and a high-frequency residual component, and rationally designing different modules for recovery respectively; by utilizing the powerful generative capability of the diffusion model, it only applies to the low-pass image, significantly accelerating the inference speed of the diffusion model; and by leveraging the advantage of the Laplacian image pyramid in preserving original information, the reconstructed clean image has higher fidelity, and has broad application prospects in image processing and other fields.

[0097] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image defogging method based on Laplacian image decomposition and diffusion model, characterized in that, The method comprises the following steps: S1: data preprocessing Make a training data set and pre-process the images; S2: Laplace image decomposition Perform Laplace image decomposition on the training images to obtain a low-pass image and a high-frequency residual part; S3: constructing an image defogging network Construct an image defogging network and train it to obtain an image defogging model; wherein the image defogging network comprises a diffusion model defogging network, a high-frequency residual recovery network and a low-frequency refining network, the low-pass image and the high-frequency residual part are respectively sent to the diffusion model defogging network and the high-frequency residual recovery network for reconstruction to obtain a recovered low-pass image and a high-frequency residual, the recovered low-pass image and the high-frequency residual are subjected to inverse transformation of the Laplace image decomposition to obtain a preliminary restored image, and finally the preliminary restored image is processed by the low-frequency refining network to obtain a final defogged image; S4: image defogging Input the foggy image to be defogged into the image defogging model to obtain a defogged image; In the step S3, the diffusion model defogging network adopts a U-Net structure design and only acts on the low-pass image, comprising a forward noise adding process and a reverse denoising process, and the specific execution process is as follows: In the paired training images, the low-pass image obtained by decomposing the haze-free image is added with noise, then the low-pass image of the foggy image is taken as a condition, and the two are spliced in the channel dimension to form a 6-channel tensor as the input of the diffusion model defogging network; the diffusion model defogging network reconstructs a clean low-pass image after iterative denoising; In step S3, the low-frequency refining network processes the preliminary restored image as follows: The obtained preliminary restored image... Haar wavelet transform was performed to extract low-frequency information A. low and high-frequency information A high Then, standard convolution is used to extract shallow features, followed by multi-scale feature extraction through three parallel residual blocks with dilated convolutions. The obtained features... It is obtained by splicing along the channel dimension. Finally, after two standard convolutions to reduce the number of channels, the enhanced low-frequency information is reconstructed. The final dehazed image can be obtained by performing an inverse wavelet transform. The entire processing procedure is defined as follows: F A = Conv(A low ); Wherein, DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform respectively, Conv() represents standard convolution, Dukated() represents the hollow convolution, the expansion rate of the hollow convolution in the three parallel residual blocks with hollow convolution is 1, 2, 3 respectively, and Concat() represents splicing in the channel dimension, is the final defogging image, is the preliminary restoration image processed by the diffusion model defogging network and the high-frequency residual recovery network, and simultaneously acts on the diffusion model defogging network and the high-frequency residual recovery network.

2. The image defogging method based on Laplacian image decomposition and diffusion model according to claim 1, characterized in that, In the step S1, the pre-processing process is to randomly crop the image size to 512x512 and perform data enhancement by horizontal flipping and random rotation.

3. The image defogging method based on Laplacian image decomposition and diffusion model according to claim 1, characterized in that, In the step S2, the original image is down-sampled to generate a series of images with gradually reduced resolution, and these images form a Gaussian pyramid; the following operations are performed on two adjacent images in the Gaussian pyramid: the next layer is up-sampled to the same size as the current layer, and then the up-sampled result is subtracted from the current layer, and the specific process definition is as follows: G i+1 = Down(G i ); L i = G i -Up(G i+1 ); where Down() and Up() denote down-sampling and up-sampling operations, respectively, G i is the low-pass image after down-sampling, L i represents the high-frequency information lost in the down-sampling process, i.e., the high-frequency residual part, L i forms a Laplacian image pyramid.

4. The image defogging method based on Laplacian image decomposition and diffusion model according to claim 1, characterized in that, The noise adding process is defined as follows: where X0is a low-pass image decomposed from the haze-free image, X t is an image with Gaussian noise added to X0, t is a time step, I is an identity matrix, denotes a Gaussian distribution, β t is a predefined variance schedule, α t = 1 - β t , denotes a random sampling of a noise ∈ t from a Gaussian distribution; In the denoising process, the diffusion model dehazing network finally obtains the clean output by iteratively predicting the noise intensity at each time step In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean value, and the prediction method of the mean value is defined as follows: wherein, represents a low-pass image of the foggy image, ∈ θ represents the noise of the parameterized network prediction.

5. The image defogging method based on Laplacian image decomposition and diffusion model according to claim 4, characterized in that, In the step S3, the high-frequency residual recovery network adopts a U-Net structure design, first uses standard convolution to extract shallow features, then performs 3 times of down-sampling and up-sampling operations, and after each up-sampling and down-sampling, a residual dense block is used for feature enhancement, and finally a standard convolution is used to restore a 3-channel output.

6. The image defogging method based on Laplacian image decomposition and diffusion model according to claim 5, characterized in that, In the step S3, when training the diffusion model defogging network and the high-frequency residual recovery network, the following comprehensive loss function is used: L total = L noise + L low + L high + L image wherein L noise is a noise prediction loss, L low is a low-frequency image reconstruction loss, acting on the diffusion model defogging network; L high is a high-frequency reconstruction loss, is a high-frequency residual predicted by the network, is its corresponding true value, acting on the high-frequency residual recovery network; L image is an image reconstruction loss, I0is a clean, fog-free image; When training the low-frequency refining network, the following loss function is used: wherein, is the final defogged image.

Images