An image dehazing method based on multi-scale encoding and decoding

Abstract

The present invention discloses an image defogging method based on multi-scale coding and decoding, comprising: S1, constructing an image dataset containing foggy images and standard fog-free images, and preprocessing the dataset; S2, constructing a multi-scale coding and decoding defogging network, and training the network using the pre-processed image dataset; S3, processing the image to be defogged using the trained multi-scale coding and decoding defogging network to obtain a defogged image. The present invention adopts a dual-branch architecture and performs multi-scale adaptive fusion of the features of the two branches, enabling the model to achieve excellent performance when using a real haze dataset of limited size; introduces brightness constraints into the combined loss function, giving positively correlated attention to regions of different concentrations, enabling the network to more effectively process non-uniform haze images; the present invention performs gamma correction on the training data of different datasets to reduce the brightness difference between the datasets, and then performs data fusion to effectively improve the defogging performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image defogging method based on multi-scale encoding and decoding. Background Art

[0002] Haze exists worldwide and is becoming more frequent due to climate change. This common atmospheric phenomenon shortens visibility, posing a significant potential risk to traffic safety. The resulting blurring and contrast loss pose significant challenges to both human observation and computer vision models, making image dehazing a crucial visual task. While researchers have proposed numerous methods to address this challenge, most current algorithms target uniform haze and are not applicable to real-world scenarios. Therefore, image dehazing algorithms for non-uniform haze scenarios present both high research value and significant challenges.

[0003] There have been many classic works in early image dehazing research. These methods can be roughly divided into traditional methods based on prior statistics and data-driven deep learning methods. He et al. pioneered the dark channel prior (DCP) algorithm, which can achieve simple and efficient image restoration. However, the failure of the dark channel prior in areas with sky or white areas often leads to halos in the restored image. Zhu et al. discovered a linear relationship between scene depth, brightness, and saturation, and proposed the color attenuation prior (CAP) dehazing algorithm. This method is fast but produces a lot of residual haze. In recent years, with the rapid development of deep learning in image processing, many dehazing methods based on CNNs (Convolutional Neural Networks) have emerged. Cai et al. first proposed the DehazeNet neural network architecture for image dehazing. This model learns the relationship between haze maps and transmittance and then uses an atmospheric scattering model to restore the image. Subsequently, Li et al. proposed a lightweight network, AOD-Net, and improved the atmospheric scattering model, integrating transmittance with atmospheric values ​​to reduce error accumulation. While these methods have improved performance over traditional approaches, the approximate physical models they rely on to estimate the scene still significantly limit reconstruction quality. With the rapid development of deep learning technology, methods such as attention mechanisms, transformers, and knowledge transfer have been proposed. These methods are often end-to-end, trained using large numbers of image pairs, and have demonstrated promising results in many fields. Chen et al. proposed a residual dehazing network (GCANet) based on smoothed dilated convolutions and a gated fusion subnetwork, addressing image artifacts. Furthermore, Qin et al. proposed a feature fusion attention network for image dehazing (FFANet) based on feature attention and pixel attention. These methods achieve impressive results for synthetic haze images, but perform poorly in real-world scenarios. This is primarily because the complexity and non-uniformity of haze distribution in real-world scenarios make data fitting extremely difficult. Furthermore, data on non-uniform haze is extremely limited, making training deep networks on such a small dataset without overfitting a significant challenge. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the image defogging method based on multi-scale encoding and decoding provided by the present invention solves the problem that the existing algorithms are difficult to apply to defogging of non-uniform haze images in real scenes.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: an image defogging method based on multi-scale encoding and decoding, comprising the following steps:

[0006] S1. Construct an image dataset containing foggy images and standard fog-free images and preprocess them;

[0007] S2. Construct a multi-scale encoding and decoding dehazing network and train it using the preprocessed image dataset. The multi-scale encoding and decoding dehazing network includes a knowledge transfer branch for knowledge supplementation, a data fitting branch for increasing network depth and improving high-frequency features, and a multi-scale feature fusion module for fusing the output features of the data fitting branch and the knowledge transfer branch.

[0008] S3. Use the trained multi-scale encoding and decoding defogging network to process the defogged image to obtain a defogged image.

[0009] Furthermore, the preprocessing in step S1 includes gamma correction processing and data enhancement processing.

[0010] Furthermore, in step S2, in the multi-scale encoding and decoding defogging network, the knowledge transfer branch includes an encoder, a decoder, and an enhancement module connected in sequence;

[0011] The encoder comprises three sequentially connected encoding blocks, each encoding block comprising a sequentially connected 7×7 convolutional layer, a layer normalization LN, two 1×1 convolutional layers, a GELU activation function, and a global response normalization GRN;

[0012] The decoder includes three decoding blocks corresponding to the encoding blocks, each of the decoding blocks includes an attention block and an upsampling block connected in sequence;

[0013] In the knowledge transfer branch, skip connections are used between encoding blocks and decoding blocks at the same level.

[0014] Furthermore, in step S2, in the multi-scale encoding and decoding dehazing network, the data fitting branch is a deep residual channel attention network, including a 3×3 convolutional layer, 4 residual channel attention groups and skip connections connected in sequence;

[0015] Each of the residual channel attention groups includes 10 residual channel attention blocks, 1 3×3 convolutional layer and skip connections connected in sequence.

[0016] Furthermore, in step S2, in the multi-scale encoding and decoding dehazing network, the multi-scale feature fusion module uses the SK fusion method to perform multi-scale connection on the output features of the knowledge transfer branch and the data fitting branch, and the reflection filling layer, the 7×7 convolution layer and the Tanh activation function are used to output the fusion features in series.

[0017] Furthermore, the expression of the SK fusion method is:

[0018]

[0019] Where attn is the attention weight, x1 is the feature map output by the knowledge transfer branch, and x2 is the feature map output by the data fitting branch. is the mapping result of x2 using a linear layer, GAP is the output of the global average pooling layer, and F MLP is the output of the layer connected in the order of Linear-ReLU-Linear, and Softmax(·) is the Softmax activation function;

[0020] The output fusion feature y is expressed as:

[0021]

[0022] Furthermore, in step S2, the loss function L when training the multi-scale encoding and decoding defogging network is joint for:

[0023]

[0024] Where, L l1 is the smooth L1 loss of the dehazing result, The smoothed L1 loss for the brightness attention generation value, L MS-SSIM is the multi-scale structural similarity loss, L perc is the perceptual loss, L adv To combat the loss, γ1, γ2, γ3, γ4, and γ5 are L l1 、 L MS-SSIM , L perc and L adv The weight of .

[0025] Furthermore, the smoothing loss L of the dehazing result l1 for:

[0026]

[0027] Where X is the total number of pixels, I gt and They are respectively the standard fog-free image and the defogging image after the defogging network, and x is the pixel point;

[0028] Smoothed L1 loss of brightness attention generation value for:

[0029]

[0030]

[0031] Where M gt The Y channel brightness value of the standard fog-free image converted to the YCbCr color space, M g represents the brightness attention value of the dehazed image in the brightness attention generation network, and γ is the difference between the true value and the predicted value;

[0032] Multi-scale structural similarity loss L MS-SSIM for:

[0033]

[0034]

[0035]

[0036] Where i is the pixel point, is the brightness comparison value, cs j (i) is the contrast and structure comparison value, j = 1, 2, 3, ..., S, S is the total number of scales, α and β j is the default parameter, μ O 、μ G ,σ O , σ G and σ OG are the mean, standard deviation, and covariance of the window centered at the i-th pixel in the dehazed image and the standard haze-free image obtained by Gaussian filtering, and C1 and C2 are constants;

[0037] Perceptual loss L perc for:

[0038]

[0039] Where, and I gt are the dehazed image and the standard haze-free image, φ j (·) is the feature map of the backbone network at layer j, (C j ,H j ,W j ) is the shape of the corresponding feature map, K is the number of features used in the perceptual loss;

[0040] Adversarial loss L adv for:

[0041]

[0042] Where D(·) is the discriminator, is the probability that the dehazed image is identified as a standard non-hazed image, and N represents the number of training samples.

[0043] The beneficial effects of the present invention are:

[0044] 1. This paper adopts a dual-branch architecture and uses the advanced ConvNeXt architecture as the backbone network of the encoder in the knowledge transfer branch. It also performs multi-scale adaptive fusion of the two branch features, enabling the model to achieve excellent performance when using a limited-size real-world haze dataset.

[0045] 2. This paper introduces a brightness constraint into the combined loss function, giving positively correlated attention to regions of different concentrations, enabling the network to more effectively process non-uniform haze images.

[0046] 3. The present invention performs gamma correction on the training data of different data sets to reduce the brightness difference between sets, and then performs data fusion to effectively improve the defogging performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the image defogging method with multi-scale encoding and decoding provided by the present invention.

[0048] Figure 2 A schematic diagram of the overall network structure provided by the present invention;

[0049] Figure 3 A schematic diagram of the coding block provided by the present invention;

[0050] Figure 4 Schematic diagram of the residual channel attention module provided by the present invention;

[0051] Figure 5 This is a schematic diagram of the brightness attention generation network structure provided by the present invention;

[0052] Figure 6 A foggy image provided by the present invention;

[0053] Figure 7 A schematic diagram of a defogging effect provided by the present invention;

[0054] Figure 8 The standard fog-free image corresponding to the foggy image provided by the present invention;

[0055] Figure 9 Another foggy image provided by the present invention;

[0056] Figure 10 A schematic diagram of another defogging effect provided by the present invention;

[0057] Figure 11 Another standard fog-free image corresponding to the foggy image provided by the present invention. DETAILED DESCRIPTION

[0058] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0059] The embodiment of the present invention provides an image defogging method based on multi-scale encoding and decoding, such as Figure 1 As shown, the following steps are included:

[0060] S1. Construct an image dataset containing foggy images and standard fog-free images and preprocess them;

[0061] S2. Construct a multi-scale encoding and decoding dehazing network and train it using the preprocessed image dataset. The multi-scale encoding and decoding dehazing network includes a knowledge transfer branch for knowledge supplementation, a data fitting branch for increasing network depth and improving high-frequency features, and a multi-scale feature fusion module for fusing the output features of the data fitting branch and the knowledge transfer branch.

[0062] S3. Use the trained multi-scale encoding and decoding defogging network to process the defogged image to obtain a defogged image.

[0063] In step S1 of the embodiment of the present invention, a dataset including foggy images and standard fog-free images can be constructed from public datasets. The image data in this embodiment is selected from DENSE-HAZE, NH-HAZE20, NH-HAZE21 and NH-HAZE23.

[0064] In step S1 of the embodiment of the present invention, preprocessing includes gamma correction and data enhancement. Gamma correction is used to reduce the average brightness difference of training data between multiple data sets, effectively improving the performance of the trained model in removing non-uniform haze. At the same time, in order to facilitate the training of the deep learning model and prevent the model from overfitting, this embodiment performs data enhancement on the training data, mainly including randomly cropping the image into image blocks of 256×256 size, and performing random rotation of 90, 180, and 270 degrees, vertical rotation, and horizontal rotation.

[0065] In step S2 of the embodiment of the present invention, a dehazing network is constructed by adopting a multi-scale deep neural network approach, and modeling is gradually performed from local to global to achieve multi-scale feature extraction and processing of the image.

[0066] In this embodiment, a multi-scale encoding and decoding defogging network is designed to learn the mapping relationship from foggy images to clear images, such as Figure 2 As shown in the figure, since each model has limitations and single-output models are susceptible to statistical and representational issues, this embodiment adopts a dual-branch structure to better leverage the strengths and weaknesses of each individual model and make more accurate decisions. The U-Net-style knowledge transfer branch supplements limited data, while the dense residual-style data fitting branch increases network depth and extracts high-frequency detail features. The multi-scale feature fusion module then adaptively fuses the dual-branch features to restore high-quality dehazed images.

[0067] In step S2 of this embodiment, in the multi-scale encoding and decoding defogging network, as shown in FIG. Figure 3 As shown, the knowledge transfer branch includes an encoder, a decoder and an enhancement module connected in sequence;

[0068] The encoder consists of three sequentially connected encoding blocks, each of which includes a 7×7 convolutional layer, a layer normalization LN (LN), two 1×1 convolutional layers, a GELU activation function, and a global response normalization GRN (GRN).

[0069] The decoder includes three decoding blocks corresponding to the encoding blocks, each of which includes an attention block and an upsampling block connected in sequence;

[0070] In the knowledge transfer branch, skip connections are used between encoding blocks and decoding blocks at the same level.

[0071] In this embodiment, in the knowledge transfer branch, the pre-trained parameters frozen on the large image classification dataset ImageNet when the encoder is initialized can effectively supplement substantial prior knowledge for the dehazing task; through jump connections between encoding blocks and decoding blocks at the same level, the hierarchical features are gradually restored and the spatial dimensions are made consistent with the input. Finally, the enhancement module is used to perform multi-scale detail enhancement on the generated features.

[0072] In step S2 of this embodiment, in the multi-scale encoding and decoding dehazing network, a deep residual channel attention network, which has excellent performance in image super-resolution tasks, is selected as the data fitting branch to extract rich texture details and represent high-frequency information. The data fitting branch in this embodiment is a deep residual channel attention network, which includes 3×3 convolutional layers, 4 residual channel attention groups, and skip connections connected in sequence; each residual channel attention group includes 10 residual channel attention blocks, 1 3×3 convolutional layer, and a skip connection connected in sequence.

[0073] In this embodiment, the structure of the residual channel attention block (RCAB) is as follows Figure 4 As shown in the figure, it uses dense residual connections to propagate each low-level feature to all subsequent high-level features, ensuring network depth while also maintaining the richness of feature information. This branch operates in full-resolution mode, maximizing fit to the current data and extracting features that differ from those of the knowledge transfer branch. The overfitting that may occur in this branch during training on limited data can also be compensated by the supplementary information extracted by the knowledge transfer branch.

[0074] In this embodiment, a deep convolutional neural network with a data fitting branch and a knowledge transfer branch is used to extract image features. The foggy image is input into a multi-scale network to extract feature maps of different receptive fields, which contain low-frequency and high-frequency information of the image. The multi-scale feature fusion module is used to adaptively spatially aggregate the feature maps that focus on different information to obtain a feature representation that is more in line with the perception characteristics of the human eye, thereby improving the image defogging effect.

[0075] In step S2 of this embodiment, in the multi-scale encoding and decoding dehazing network, the multi-scale feature fusion module uses the SK fusion method to perform multi-scale connection on the output features of the knowledge transfer branch and the data fitting branch, and outputs the fused features in series through the reflection filling layer, the 7×7 convolution layer and the Tanh activation function.

[0076] In this embodiment, the expression of the SK fusion mode is:

[0077]

[0078] Where attn is the attention weight, x1 is the feature map output by the knowledge transfer branch, and x2 is the feature map output by the data fitting branch. is the mapping result of x2 using a linear layer, GAP is the output of the global average pooling layer, and F MLP is the output of the layer connected in the order of Linear-ReLU-Linear, Softmax(·) is the Softmax activation function, and its expression is:

[0079]

[0080] The output fusion feature y is expressed as:

[0081]

[0082] In step S2 of this embodiment, the loss function L when training the multi-scale encoding and decoding defogging network is joint for:

[0083]

[0084] Where, L l1 is the smooth L1 loss of the dehazing result, The smoothed L1 loss for the brightness attention generation value, L MS-SSIM is the multi-scale structural similarity loss, L perc is the perceptual loss, L adv To combat the loss, γ1, γ2, γ3, γ4, and γ5 are L l1 、 L MS-SSIM , L perc and L adv In this embodiment, the weights are set to 1, 0.3, 0.5, 0.01 and 0.0005 respectively.

[0085] In this embodiment, the smooth L1 loss includes not only the measurement of the defogging result but also the measurement of the generated brightness attention value. Figure 5 As shown, because non-uniform haze will significantly increase the brightness of the blocked area (except the sky area), in order to pay more attention to the recovery of areas with large brightness changes, that is, dense haze areas, a brightness attention generation network is introduced in this embodiment to output a brightness attention map.

[0086] Specifically, the smoothness loss L of the dehazing result is l1 and smoothed L1 loss of brightness attention generation value They are:

[0087]

[0088]

[0089]

[0090] Where X is the total number of pixels, I gt and They are respectively the standard fog-free image and the defogging image after the defogging network, x is the pixel point, M gt The Y channel brightness value of the standard fog-free image converted to the YCbCr color space, M g represents the brightness attention value of the dehazed image in the brightness attention generation network, and γ is the difference between the true value and the predicted value.

[0091] In this embodiment, the scale structure similarity loss is based on the assumption that the human eye is suitable for extracting structural information. Taking it as part of the loss function can improve the perceptual quality of the image. MS-SSIM It can be expressed as:

[0092]

[0093]

[0094]

[0095] Where i is the pixel point, is the brightness comparison value, cs j (i) is the contrast and structure comparison value, j = 1, 2, 3, ..., S, S is the total number of scales, α and β j is the default parameter, μ O 、μ G ,σ O , σ G and σ OG are the mean, standard deviation, and covariance of the window centered at the i-th pixel in the dehazed image and the standard haze-free image obtained by Gaussian filtering, and C1 and C2 are constants.

[0096] In this embodiment, the perceptual loss can improve the similarity between the defogging result and the standard non-fogging image in the feature space by supervising high-level features. The VGG-16 model pre-trained on the ImageNet dataset is introduced as the loss network φ, and the perceptual loss L is used as the loss network φ. perc Expressed as:

[0097]

[0098] Where, and I gt are the dehazed image and the standard haze-free image, φ j (·) is the feature map of the backbone network at layer j, (C j ,H j ,W j ) is the shape of the corresponding feature map, K is the number of features used in the perceptual loss;

[0099] In this embodiment, when there is only a small-scale non-uniform haze dataset, the adversarial loss is used to reduce the risk of insufficient supervision caused by pixel loss alone; the adversarial loss L established with the help of the generator loss formula is adv Expressed as:

[0100]

[0101] Where D(·) is the discriminator, is the probability that the dehazed image is identified as a standard non-hazed image, and N represents the number of training samples.

[0102] In this embodiment, based on the above-mentioned image defogging method, Figure 6 and Figure 9 Perform image dehazing processing to obtain Figure 7 and Figure 10 The corresponding defogging effect diagram, Figure 8 and Figure 11 They are Figure 6 and Figure 9 The corresponding standard haze-free image.

[0103] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0104] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. An image defogging method based on multi-scale encoding and decoding, characterized in that: The following steps are involved: S1. Construct an image dataset containing foggy images and standard fog-free images and preprocess them; S2. Construct a multi-scale encoding and decoding dehazing network and train it using the preprocessed image dataset. The multi-scale encoding and decoding dehazing network includes a knowledge transfer branch for knowledge supplementation, a data fitting branch for increasing network depth and improving high-frequency features, and a multi-scale feature fusion module for fusing the output features of the data fitting branch and the knowledge transfer branch. S3, using the trained multi-scale encoding and decoding defogging network to process the defogging image to obtain a defogging image; In step S2, in the multi-scale encoding and decoding dehazing network, the multi-scale feature fusion module uses the SK fusion method to perform multi-scale connection on the output features of the knowledge transfer branch and the data fitting branch, and connects the fused features in series through the reflection padding layer, the 7×7 convolution layer and the Tanh activation function; The expression of the SK fusion method is: Where, is the attention weight, is the feature map output by the knowledge transfer branch, The feature map output by the data fitting branch, To use a linear layer pair The mapping result is is the output of the global average pooling layer, To connect the output of the layer in the order of Linear-ReLU-Linear, is the Softmax activation function; Output fusion features Expressed as: 。 2. The image defogging method based on multi-scale encoding and decoding according to claim 1, characterized in that: The pre-processing in step S1 includes gamma correction processing and data enhancement processing.

3. The image defogging method based on multi-scale encoding and decoding according to claim 1, characterized in that: In step S2, in the multi-scale encoding and decoding defogging network, the knowledge transfer branch includes an encoder, a decoder, and an enhancement module connected in sequence; The encoder includes three sequentially connected encoding blocks, each encoding block includes a 7×7 convolutional layer, a layer normalization LN, two 1×1 convolutional layers, a GELU activation function and a global response normalization GRN connected in sequence; The decoder includes three decoding blocks corresponding to the encoding blocks, each of the decoding blocks includes an attention block and an upsampling block connected in sequence; In the knowledge transfer branch, skip connections are used between encoding blocks and decoding blocks at the same level.

4. The image defogging method based on multi-scale encoding and decoding according to claim 1, characterized in that: In step S2, in the multi-scale encoding and decoding dehazing network, the data fitting branch is a deep residual channel attention network, including a 3×3 convolutional layer, 4 residual channel attention groups and skip connections connected in sequence; Each of the residual channel attention groups includes 10 residual channel attention blocks, 1 3×3 convolutional layer and skip connections connected in sequence.

5. The image defogging method based on multi-scale encoding and decoding according to claim 1, characterized in that: In step S2, the loss function when training the multi-scale encoding and decoding defogging network is for: Where, is the smooth L1 loss of the dehazing result, A smoothed L1 loss for brightness attention generation, is the multi-scale structural similarity loss, is the perceptual loss, To combat losses, They are 、 、 、 and The weight of .

6. The image defogging method based on multi-scale encoding and decoding according to claim 5, characterized in that: The smoothing loss of the dehazing result for: Where, X is the total number of pixels, and They are respectively the standard fog-free image and the defogging image after the defogging network. x is a pixel; Smoothed L1 loss of brightness attention generation value for: Where, Convert the standard fog-free image to the Y channel brightness value of the YCbCr color space. represents the brightness attention value of the dehazed image in the brightness attention generation network, is the difference between the true value and the predicted value; Multi-scale structural similarity loss for: Where, i is a pixel, is the brightness comparison value, is the contrast and structure comparison value, , S is the total number of scales, and is the default parameter, 、 , 、 and The dehazed image and the standard haze-free image are i The mean, standard deviation and covariance of the window centered on the pixel point are obtained by Gaussian filtering. and is a constant; Perceptual loss for: Where, and are the dehazed image and the standard haze-free image, For the backbone network j The feature map of the layer, is the shape of the corresponding feature map, K is the number of features used in the perceptual loss; Fighting Losses for: Where, is the discriminator, is the probability that the dehazed image is identified as a standard non-hazed image, N Indicates the number of training samples.

Images