Image defogging method based on Laplace image decomposition and diffusion model

Abstract

The invention discloses an image defogging method based on a Laplace image decomposition and diffusion model, which belongs to the technical field of image processing, and comprises the following steps: S1, data preprocessing; s2, decomposing a Laplace image; s3, constructing an image defogging network; and S4, image defogging. According to the invention, a foggy image is decomposed into a low-pass image and a high-frequency residual error part, and different modules are reasonably designed for respective recovery, so that the defogging difficulty is reduced; the strong generation capability of the diffusion model is utilized, the diffusion model only acts on a low-pass image, and the reasoning speed of the diffusion model is remarkably increased; and by utilizing the advantage that the Laplacian image pyramid retains original information, the reconstructed clean image is enabled to have higher fidelity, and the method has wide application prospects in the fields of image processing and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method based on Laplace image decomposition and diffusion model. Background Art

[0002] With the continuous development of computer vision and image processing technology, image dehazing technology has become an important research direction in many practical applications. Haze weather can cause problems such as reduced image contrast and color distortion, affecting the visual effect of the image and subsequent automated processing, such as target recognition, scene understanding, and navigation systems. Traditional image dehazing methods rely on statistically discovered prior knowledge to predict transmission maps and global atmospheric light, and finally restore haze-free images in combination with atmospheric scattering models. However, these prior knowledge does not always fit various haze scenes and can easily cause problems such as color distortion. In recent years, the development of deep learning technology has provided new ideas for image dehazing. Researchers have begun to directly restore clean images in an end-to-end manner by designing complex convolutional neural networks (CNNs) or Transformer networks. However, these methods usually have limited performance in dealing with complex haze scenes, especially in high-concentration haze or non-uniformly distributed haze environments, where the dehazing effect is difficult to achieve as expected.

[0003] After searching, the Chinese invention patent with announcement number CN109712083B discloses a single image defogging method based on convolutional neural network. Its network design includes a shallow neural network model and a deep neural network model. The shallow network model is used to extract and fuse the features of the color space of the foggy image to obtain the scene depth map of the foggy image; the deep network model performs multi-scale mapping, pooling, convolution and other operations on the scene depth map, and outputs the transmittance map of the foggy image. Finally, the fog-free image is restored based on the atmospheric scattering model. However, the distribution of haze in real scenes is often more complex, deviating from the ideal data assumptions of the atmospheric scattering model, resulting in poor defogging effect.

[0004] The Chinese invention patent with announcement number CN113066028B discloses an image dehazing method based on a Transformer deep neural network, which utilizes the inherent global feature extraction capability of the Transformer and assists network training by reconstructing foggy images. However, the Transformer network needs to rely on larger and higher-quality data for training. Paired foggy / fog-free images in real scenes are difficult to obtain, and the training set size is small, which limits the performance of the network model and produces serious color deviation.

[0005] Diffusion models have gradually attracted the attention of researchers due to their excellent performance in high-quality image generation and denoising. Diffusion models can gradually restore detailed information in images by simulating the gradual evolution of data, especially when dealing with noise or degradation problems. Compared with other deep learning models, diffusion models show stronger adaptability and robustness, effectively solving the shortcomings of traditional dehazing methods in the face of complex haze conditions. However, traditional diffusion models have slow inference speed and require a lot of computing resources and time overhead.

[0006] Based on the above analysis, for the more complex haze distribution in real scenes, the existing technology requires an image dehazing method that is highly adaptable and saves more computing resources. To this end, an image dehazing method based on Laplace image decomposition and diffusion model is proposed. Summary of the invention

[0007] The technical problem to be solved by the present invention is: how to overcome the limitations of traditional deep neural networks in the face of limited feature extraction capabilities of complex haze scenes and slow diffusion model inference speed, and then solve problems such as insufficient dehazing and color distortion, and provide an image dehazing method based on Laplace image decomposition and diffusion model.

[0008] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0009] S1: Data preprocessing

[0010] Create a training data set and preprocess the images;

[0011] S2: Laplacian image decomposition

[0012] Perform Laplace image decomposition on the training image to obtain a low-pass image and a high-frequency residual part;

[0013] S3: Building an image dehazing network

[0014] Construct an image defogging network and train it to obtain an image defogging model; wherein the image defogging network includes a diffusion model defogging network, a high-frequency residual recovery network and a low-frequency refinement network. The low-pass image and the high-frequency residual part are respectively sent to the diffusion model defogging network and the high-frequency residual recovery network for reconstruction to obtain the restored low-pass image and high-frequency residual. Starting from the low-pass image, a preliminary restored image can be obtained through the inverse transformation of the Laplace image decomposition. Finally, the preliminary restored image is processed by the low-frequency refinement network to obtain the final defogging image.

[0015] S4: Image Dehazing

[0016] The foggy image to be defogged is input into the image defogging model to obtain the defogged image.

[0017] Furthermore, in step S1, the preprocessing process is to randomly crop the image size to 512×512, and perform horizontal flipping and random rotation for data enhancement.

[0018] Furthermore, in step S2, the original image is downsampled to generate a series of images with gradually reduced resolutions, and these images form a Gaussian pyramid; the following operations are performed on two adjacent layers of images in the Gaussian pyramid: the next layer is upsampled to the same size as the current layer, and then the upsampled result is subtracted from the current layer. The specific process is defined as follows:

[0019] G i+1 = Down(G i );

[0020] L i =G i -Up(G i+1 );

[0021] Among them, Down() and Up() represent downsampling and upsampling operations respectively, G i is the low-pass image after downsampling, L i Represents the high-frequency information lost during the downsampling process, that is, the high-frequency residual part, L i Form a Laplacian image pyramid.

[0022] Furthermore, in step S3, the diffusion model defogging network adopts a U-Net structure design and acts only on the low-pass image to achieve overall haze removal, including a forward denoising process and a reverse denoising process. The specific execution process is as follows:

[0023] In the paired training images, the low-pass image obtained by decomposing the haze-free image is denoised, and then the low-pass image of the haze image is used as a condition. The two are spliced ​​in the channel dimension to form a 6-channel tensor as the input of the diffusion model defogging network; under the guidance of haze characteristics, the diffusion model defogging network can reconstruct a clean low-pass image after iterative denoising.

[0024] Furthermore, the noise adding process is defined as follows:

[0025]

[0026]

[0027] Among them, X0 is the low-pass image obtained by decomposing the fog-free image, X t is the image after adding Gaussian noise to X0, t is the time step, I is the unit matrix, represents Gaussian distribution, β t is the predefined variance schedule, α t =1-β t , Represents a random sampling of a noise ∈ from a Gaussian distribution t ;

[0028] In the denoising process, the diffusion model dehazing network iteratively predicts the noise intensity at each time step and finally obtains a clean output. In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean. The prediction method of the mean is defined as follows:

[0029]

[0030] in, Represents the low-pass image of the foggy image, ∈ θ represents the noise in the parameterized network predictions.

[0031] Furthermore, in step S3, the high-frequency residual recovery network adopts a U-Net structure design, first using standard convolution to perform shallow feature extraction, then undergoing three downsampling and upsampling operations, after each upsampling and downsampling, the residual dense block is used for feature enhancement, and finally the standard convolution is used to restore the output to a three-channel output.

[0032] Furthermore, in step S3, when the diffusion model dehazing network and the high-frequency residual recovery network are trained, the following comprehensive loss function is used:

[0033] L total =L noise +L low +L high +L image

[0034] Among them, L noise is the noise prediction loss, L low is the low-pass image reconstruction loss, Acts on the diffusion model dehazing network; L high is the high-frequency reconstruction loss, is the high-frequency residual predicted by the network, is its corresponding true value, acting on the high-frequency residual recovery network; L image is the image reconstruction loss, I0 is a clean, fog-free image. It is the initial restored image after being processed by the diffusion model dehazing network and the high-frequency residual restoration network, and acts on the diffusion model dehazing network and the high-frequency residual restoration network at the same time;

[0035] When training the low-frequency refining network, the following loss function is used:

[0036]

[0037] in, The final dehazed image.

[0038] Furthermore, in step S3, the specific processing process of the low-frequency refining network on the preliminary restored image is as follows: Haar wavelet transform is performed on the obtained preliminary restored image to extract the low-frequency information A low , and then use standard convolution to extract shallow features, and then perform multi-scale feature extraction through 3 parallel residual blocks with dilated convolutions. Concatenate in the channel dimension to get Finally, two standard convolutions are performed to reduce the number of channels and reconstruct the enhanced low-frequency information. The final dehazed image can be obtained by inverse wavelet transform. The entire processing process is defined as follows:

[0039]

[0040] F A =Conv(A low );

[0041]

[0042]

[0043]

[0044]

[0045] Among them, DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform respectively, Conv() represents standard convolution, Dilated() represents dilated convolution, the dilation rates of dilated convolution in the three parallel residual blocks with dilated convolution are 1, 2, and 3 respectively, and Concat() represents splicing in the channel dimension. The final dehazed image.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] (1) The present invention decomposes a foggy image into a low-pass image and a high-frequency residual part, and rationally designs different modules to perform restoration respectively, thereby reducing the difficulty of defogging.

[0048] (2) The present invention utilizes the powerful generation capability of the diffusion model and only acts on low-pass images, which significantly speeds up the inference speed of the diffusion model.

[0049] (3) The present invention utilizes the advantage of the Laplacian image pyramid in preserving the original information, so that the reconstructed clean image has higher fidelity, and has broad application prospects in the fields of image processing and so on. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of an image defogging method based on Laplace image decomposition and diffusion model of the present invention;

[0051] Figure 2 It is a schematic diagram of the process of Laplace image decomposition in the present invention;

[0052] Figure 3 It is a structural schematic diagram of the image defogging network constructed in the present invention, wherein the image defogging network includes a diffusion model defogging network, a high-frequency residual recovery network and a low-frequency refining network;

[0053] Figure 4 It is a structural schematic diagram of the high-frequency residual recovery network in the present invention;

[0054] Figure 5 It is a structural schematic diagram of the low-frequency refining network in the present invention. DETAILED DESCRIPTION

[0055] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0056] like Figure 1 As shown, an image defogging method based on Laplace image decomposition and diffusion model in this embodiment specifically includes the following steps:

[0057] Step 1: Create a training data set and preprocess the images: First, randomly crop the images to a size of 512×512, and perform horizontal flipping and random rotation for data augmentation.

[0058] Step 2: Perform Laplace image decomposition on the training image to obtain a low-resolution low-pass image and a high-frequency residual part. The specific implementation is as follows: Figure 2 As shown:

[0059] In this embodiment, the scale of Laplacian image decomposition is 2. First, the original image is downsampled to generate a series of images with gradually decreasing resolutions, which form a Gaussian pyramid. The following operations are performed on two adjacent layers of images in the Gaussian pyramid: the next layer is upsampled to the same size as the current layer, and then the upsampled result is subtracted from the current layer. This process retains the high-frequency detail information lost during downsampling, which can ensure that there is no information loss during the decomposition process. The specific process can be defined as:

[0060] G i+1 =Down(G i );

[0061] L i =G i -Up(G i+1 );

[0062] Among them, Down() and Up() represent downsampling and upsampling operations respectively, G i is the low-pass image after downsampling, L i Represents the high-frequency information lost during the downsampling process, that is, the high-frequency residual part, L i Form a Laplacian image pyramid.

[0063] Step 3: Construct a diffusion model defogging network and a high-frequency residual restoration network, and send the low-pass image and the high-frequency residual part to the diffusion model defogging network and the high-frequency residual restoration network for reconstruction. The execution process of the entire network is as follows: Figure 3 As shown, the specific structure of the high-frequency residual recovery network is as follows Figure 4 shown.

[0064] In this embodiment, the specific processing process of step 3 is as follows:

[0065] Step 3-1: The constructed diffusion model dehazing network only acts on the low-pass image to achieve the overall haze removal, including the forward denoising process and the reverse denoising process. The specific execution process is as follows:

[0066] In the paired training images, the low-pass image obtained by decomposing the haze-free image is denoised, and then the low-pass image of the haze image is used as a condition. The two are spliced ​​in the channel dimension to form a 6-channel tensor as the input of the diffusion model defogging network. Under the guidance of haze characteristics, the diffusion model defogging network can reconstruct a clean low-pass image after iterative denoising. The denoising process is defined as:

[0067]

[0068]

[0069] Among them, X0 is the low-pass image obtained by decomposing the fog-free image, X t is the image after adding Gaussian noise to X0, t is the time step, I is the unit matrix, represents Gaussian distribution, β t is the predefined variance schedule, α t =1-β t , Represents a random sampling of a noise ∈ from a Gaussian distribution t ;

[0070] During the denoising process, the diffusion model dehazing network iteratively predicts the noise intensity at each time step and finally obtains a clean output. In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean, which can be defined as:

[0071]

[0072] in, Represents the low-pass image of the foggy image, ∈ θ represents the noise in the parameterized network predictions.

[0073] Step 3-2: The high-frequency residual recovery network adopts the U-Net structure design. First, standard convolution is used to extract shallow features, and then three downsampling and upsampling operations are performed. After each upsampling and downsampling, the residual dense block (RDB) is used for feature enhancement, and finally the standard convolution is used to restore the 3-channel output.

[0074] Step 3-3: Based on the restored low-pass image and high-frequency residual, starting from the low-pass image, a preliminary restored image can be obtained through the inverse transform of the Laplace image decomposition.

[0075] Step 4: Use the pytorch deep learning framework to train the network model built in step 3. The optimizer selects the Adam optimizer. For the diffusion model dehazing network acting on the low-pass image, in addition to using the noise prediction loss, the low-pass image reconstruction loss is also used for constraints, which can be defined as:

[0076]

[0077]

[0078] in, is the low-pass image of the sampled prediction.

[0079] For the high-frequency residual recovery network, the high-frequency reconstruction loss is used for constraint, which can be defined as:

[0080]

[0081] in, is the high-frequency residual predicted by the network, is its corresponding true value, and the value of k in this embodiment is 2. In addition, the image reconstruction loss is used to constrain the two networks, which is defined as:

[0082]

[0083] Among them, I0 is a clean fog-free image, is the initial restored image after being processed by the diffusion model dehazing network and the high-frequency residual restoration network. The total loss function is the sum of the above loss functions and is defined as:

[0084] L total =L noise +L low +L high +L image .

[0085] Step 5: Construct a low-frequency refining network and apply it to the initial restored image obtained in step 3. Figure 5 As shown, the initial restored image obtained in step 3 is first subjected to Haar wavelet transform to extract the low-frequency information A low , and then use standard convolution to extract shallow features, and then perform multi-scale feature extraction through 3 parallel residual blocks with dilated convolutions. Concatenate in the channel dimension to get Finally, two standard convolutions are performed to reduce the number of channels and reconstruct the enhanced low-frequency information. The final dehazed image can be obtained by inverse wavelet transform. The whole process is defined as:

[0086]

[0087] F A =Conv(A low );

[0088]

[0089]

[0090]

[0091]

[0092] Among them, DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform respectively, Conv() represents standard convolution, Dilated() represents dilated convolution, the dilation rates of dilated convolution in Branch1, Branch2, and Branch3 are 1, 2, and 3 respectively, and Concat() represents splicing in the channel dimension. The final dehazed image.

[0093] Step 6: Train the low-frequency refinement network to obtain weight parameters. The training environment settings are consistent with step 4. The loss function is defined as:

[0094]

[0095] Step 7: Take a foggy image as the input of the model and use the weight parameters learned in steps 4 and 6 to reconstruct a high-fidelity fog-free image as the output.

[0096] In summary, the image defogging method based on Laplace image decomposition and diffusion model in the above embodiment reduces the difficulty of defogging by decomposing a foggy image into a low-pass image and a high-frequency residual part, and reasonably designs different modules to restore them respectively; utilizes the powerful generation ability of the diffusion model and only acts on the low-pass image, which significantly speeds up the reasoning speed of the diffusion model; utilizes the advantage of the Laplace image pyramid in retaining the original information, so that the reconstructed clean image has higher fidelity, and has broad application prospects in image processing and other fields.

[0097] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image dehazing method based on Laplace image decomposition and diffusion model, characterized in that: The following steps are involved: S1: Data preprocessing Create a training data set and preprocess the images; S2: Laplacian image decomposition Perform Laplace image decomposition on the training image to obtain a low-pass image and a high-frequency residual part; S3: Building an image dehazing network Construct an image defogging network and train it to obtain an image defogging model; wherein the image defogging network includes a diffusion model defogging network, a high-frequency residual recovery network and a low-frequency refinement network. The low-pass image and the high-frequency residual part are respectively sent to the diffusion model defogging network and the high-frequency residual recovery network for reconstruction to obtain the restored low-pass image and high-frequency residual. Starting from the low-pass image, a preliminary restored image can be obtained through the inverse transformation of the Laplace image decomposition. Finally, the preliminary restored image is processed by the low-frequency refinement network to obtain the final defogging image. S4: Image Dehazing The foggy image to be defogged is input into the image defogging model to obtain the defogged image.

2. The image defogging method based on Laplace image decomposition and diffusion model according to claim 1, characterized in that: In step S1, the preprocessing process is to randomly crop the image size to 512×512, and perform horizontal flipping and random rotation for data enhancement.

3. The image defogging method based on Laplace image decomposition and diffusion model according to claim 1, characterized in that: In step S2, the original image is downsampled to generate a series of images with gradually reduced resolutions, and these images form a Gaussian pyramid; the following operations are performed on two adjacent layers of images in the Gaussian pyramid: the next layer is upsampled to the same size as the current layer, and then the upsampled result is subtracted from the current layer. The specific process is defined as follows: G i+1 =Down(G i ); L i =G i -Up(G i+1 ); Among them, Down() and Up() represent downsampling and upsampling operations respectively, G i is the low-pass image after downsampling, L i Represents the high-frequency information lost during the downsampling process, that is, the high-frequency residual part, L i Form a Laplacian image pyramid.

4. The image defogging method based on Laplace image decomposition and diffusion model according to claim 3, characterized in that: In step S3, the diffusion model defogging network adopts a U-Net structure design and acts only on the low-pass image to achieve overall haze removal, including a forward denoising process and a reverse denoising process. The specific execution process is as follows: In the paired training images, the low-pass image obtained by decomposing the haze-free image is denoised, and then the low-pass image of the haze image is used as a condition. The two are spliced ​​in the channel dimension to form a 6-channel tensor as the input of the diffusion model defogging network; under the guidance of haze characteristics, the diffusion model defogging network can reconstruct a clean low-pass image after iterative denoising.

5. The image defogging method based on Laplace image decomposition and diffusion model according to claim 4, characterized in that: The noise adding process is defined as follows: Among them, X0 is the low-pass image obtained by decomposing the fog-free image, X t is the image after adding Gaussian noise to X0, t is the time step, I is the unit matrix, represents Gaussian distribution, β t is the predefined variance schedule, α t =1-β t , Represents a random sampling of a noise ∈ from a Gaussian distribution t ; In the denoising process, the diffusion model dehazing network iteratively predicts the noise intensity at each time step and finally obtains a clean output. In this process, the variance of the noise is a fixed constant, and the diffusion model dehazing network only needs to predict the mean. The prediction method of the mean is defined as follows: in, Represents the low-pass image of the foggy image, ∈ θ represents the noise in the parameterized network predictions.

6. The image defogging method based on Laplace image decomposition and diffusion model according to claim 5, characterized in that: In step S3, the high-frequency residual recovery network adopts a U-Net structure design. First, standard convolution is used to extract shallow features, and then three downsampling and upsampling operations are performed. After each upsampling and downsampling, the residual dense block is used to enhance the features, and finally the standard convolution is used to restore the output to a three-channel output.

7. The image defogging method based on Laplace image decomposition and diffusion model according to claim 6, characterized in that: In step S3, when training the diffusion model dehazing network and the high-frequency residual recovery network, the following comprehensive loss function is used: L total =L noise +L low +L high +L image Among them, L noise is the noise prediction loss, L low is the low-pass image reconstruction loss, Acts on the diffusion model dehazing network; L high is the high-frequency reconstruction loss, is the high-frequency residual predicted by the network, is its corresponding true value, acting on the high-frequency residual recovery network; L image is the image reconstruction loss, I0 is a clean, fog-free image. It is the initial restored image after being processed by the diffusion model dehazing network and the high-frequency residual restoration network, and acts on the diffusion model dehazing network and the high-frequency residual restoration network at the same time; When training the low-frequency refining network, the following loss function is used: in, The final dehazed image.

8. The image defogging method based on Laplace image decomposition and diffusion model according to claim 7, characterized in that: In step S3, the specific processing process of the low-frequency refining network on the preliminary restored image is as follows: Haar wavelet transform is performed on the obtained preliminary restored image to extract the low-frequency information A low , and then use standard convolution to extract shallow features, and then perform multi-scale feature extraction through 3 parallel residual blocks with dilated convolutions. Concatenate in the channel dimension to get Finally, two standard convolutions are performed to reduce the number of channels and reconstruct the enhanced low-frequency information. The final dehazed image can be obtained by inverse wavelet transform. The entire processing process is defined as follows: F A =Conv(A low ); Among them, DWT() and IWT() represent Haar wavelet transform and inverse wavelet transform respectively, Conv() represents standard convolution, Dilated() represents dilated convolution, the dilation rates of dilated convolution in the three parallel residual blocks with dilated convolution are 1, 2, and 3 respectively, and Concat() represents splicing in the channel dimension. The final dehazed image.

Images