Image defogging method based on improved DEA-Net

Abstract

The invention discloses an image defogging method based on improved DEA-Net. The method comprises the following steps: (1) obtaining real image foggy data of an open source website and a corresponding clear image as a training set and a test set; (2) the training set images are cut and then preprocessed; (3) constructing a deep learning model, wherein the model comprises a coding part, a feature transformation part and a decoding part; (4) inputting training image data in a training set into the created network model, and training by using a GPU (Graphic Processing Unit) to obtain model parameters with optimal accuracy; (5) taking the test image as input to obtain a defogged image; according to the method, the detail texture features of the original image can be better reserved in the defogging process, and distortion caused by model defogging is reduced. Meanwhile, the structure of the model is simplified, and model complexity and training time are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method based on improved DEA-Net. Background Art

[0002] Image defogging is often considered a necessary step in industrial applications before performing advanced vision tasks. For example, through defogging technology, autonomous driving systems or driver assistance systems can obtain clearer road information on foggy days, accurately identify vehicles, pedestrians and obstacles ahead, improve driving safety and reduce accidents. This makes image defogging one of the representative tasks of low-level vision and has attracted more and more attention from researchers. Traditional image defogging methods are mainly divided into two categories: physical model-based methods and image enhancement techniques. The core idea of ​​the image enhancement-based defogging algorithm is to minimize the noise in the image and enhance the image contrast to restore a clear, fog-free image. The physical model-based defogging method assumes an atmospheric scattering model and restores the original clarity of the image by establishing a physical relationship. The physical model-based defogging method has a more theoretical basis, but it performs poorly in complex scenes and has a large amount of calculation. Image enhancement technology is relatively simple, but it is easy to cause detail loss and artifact generation, and cannot accurately restore the true appearance of fog-free images. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide an image dehazing method based on an improved DEA-Net to solve the problems of poor performance in complex scenes, large amount of calculation, loss of details and artifact generation, and inability to accurately restore the true appearance of a haze-free image.

[0004] Technical solution: The image defogging method based on improved DEA-Net described in the present invention comprises the following steps:

[0005] (1) Obtain foggy image data and corresponding clear images from open source websites as training and test sets;

[0006] (2) Cropping and preprocessing the training set images;

[0007] (3) Build a deep learning model, which includes an encoding part, a feature transformation part, and a decoding part;

[0008] (4) Input the training image data in the training set into the created network model and use the GPU for training to obtain the model parameters with the best accuracy;

[0009] (5) Take the test image as input and obtain the dehazed image.

[0010] Furthermore, in step (2), the training set is cropped and the crop size is set to 1024 × 1024. Then, the training set images are preprocessed before being input into the deep learning model for training: randomly cropped to 256 × 256 size and randomly rotated 90 degrees.

[0011] Furthermore, in step (3), the encoding part of the model is mainly responsible for downsampling and feature extraction of the input image, and is composed of a downsampling module and a convolution block; wherein the convolution block for feature extraction utilizes the detail enhancement block DEB in DEA-Net, and the downsampling block utilizes the WD downsampling block; wherein the WD downsampling block is a wavelet downsampling block.

[0012] Furthermore, in step (3), the wavelet downsampling block is as follows: First, let the input feature map X i After discrete wavelet transform, a low-frequency component LL and three high-frequency components are obtained: horizontal high-frequency component HL, vertical high-frequency component LH, and diagonal high-frequency component HH; then the low-frequency component and the three high-frequency components are spliced ​​along the channel dimension to form a new feature map; the new feature map is subjected to 1×1 convolution, batch normalization, activation function Relu, and 3×3 convolution to obtain the downsampled feature map X o .

[0013] Furthermore, in step (3), the feature transformation part of the model is constructed. The feature transformation part is located at a deeper level of the model and is used to enhance feature expression and gradually remove the interference of fog on the image. The feature transformation part is composed of a convolution block and a feature fusion module. The convolution block adopts the attention block DEAB in DEA-Net. The feature fusion module adopts SA-Fusion, which uses an attention mechanism to adaptively select the most useful features from multiple feature maps for fusion.

[0014] Furthermore, in step (3), the SA-Fusion formula is as follows:

[0015] {a1,a2}=Split(Softmax(F mlp (AAP(F low +F high ))))

[0016] where F low is a low-level feature, F high is a high-level feature, AAP is adaptive average pooling, F mlp is the multilayer perceptron in the middle of the fusion module. Multiply the obtained weights with the corresponding feature maps and then add the two results to get the final output:

[0017] F o =F low ×a1+F high×a2

[0018] Furthermore, in step (3), the decoding part of the model is composed of a convolution block, a feature fusion module and an upsampling module; the convolution block adopts a detail enhancement convolution block DEB, and the upsampling block adopts a BTU upsampling module that fuses bilinear interpolation and transposed convolution. In feature fusion, the features after the downsampling operation and the corresponding features before the upsampling operation are fused through the SA-Fusion fusion module; a global residual is added at the end of the network to retain the original input information.

[0019] Furthermore, step (4) is as follows: the number of training iterations is 40,000, and the initial learning rate is 1e -4 , the cosine annealing strategy is used to adjust the learning rate from the initial value to 1e -6 , Adam optimizer is used with a batch-size of 8; the test image is taken as input to obtain the dehazed image, and the structural similarity and peak signal-to-noise ratio are used as evaluation indicators to evaluate the dehazing effect.

[0020] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the image defogging methods based on the improved DEA-Net is implemented.

[0021] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, any one of the image defogging methods based on the improved DEA-Net is implemented.

[0022] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it improves the defogging effect of the original model on the real image data set, can better preserve the detailed texture features of the original image during the defogging process, and reduces the distortion caused by the defogging of the model. At the same time, it simplifies the structure of the model, reduces the model complexity and training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic flow chart of the method of the present invention;

[0024] Figure 2 is a schematic diagram of an improved deep learning model of the present invention;

[0025] Figure 3 is a schematic diagram of a WD downsampling module of the present invention;

[0026] Figure 4 is a schematic diagram of the SA-Fusion module of the present invention;

[0027] Figure 5is a schematic diagram of a BTU upsampling module of the present invention;

[0028] Figure 6 It is a comparison chart of the defogging results of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0030] like Figure 1 As shown, the embodiment of the present invention provides an image defogging method based on improved DEA-Net, comprising the following steps:

[0031] Step 1: First, divide the real image defogging dataset. The dataset contains 45 images in total, including foggy images and their corresponding clear images. The first 35 images are used as training sets, the last 5 images are used as test sets, and the middle 5 images are discarded. Then the training set is cropped, and the crop size crop_size is set to 1024×1024. The stride of each cropping is crop_size / 3, that is, there is a 2 / 3 overlap between the cropped blocks. The number of steps for vertical cropping can be obtained:

[0032]

[0033] Number of horizontal cropping steps:

[0034]

[0035] If the height h of the image is greater than crop_size, it needs to be cropped multiple times, each time moving stride steps in the vertical direction; if the image height is less than or equal to crop_size, it only needs to be cropped once, and the same applies to the horizontal direction. Through cropping, the larger image in the training set is divided into multiple smaller overlapping image blocks that can cover the entire original image. After cutting, the number of training set pictures becomes 2520, and these pictures are used as training sets in actual training. Then, before inputting the training set pictures into the deep learning model for training, they are preprocessed again. Randomly crop them into 256×256 size and perform a random 90-degree rotation. Through the cutting process, the model learns more high-resolution details from different areas during the training process, helping it to better process large images during reasoning. The random cropping of large images by the original model may cause important information to be lost and reduce the model effect. Through the cutting method of the present invention, the training data of small images is more concentrated, which is conducive to model learning.

[0036] Step 2: After data preprocessing, build a deep learning model. The model consists of three parts: encoder, feature transformation (the bottleneck in the middle), and decoder. Figure 2As shown, it includes the newly designed WD module, SA-Fusion module and BTU module. We deploy the detail enhancement attention block DEAB proposed by DEA-Net in the feature conversion part, and the detail enhancement block DEB in the rest, and the number of deployments N1 = 4.

[0037] Step 3: The encoding part of the model is mainly responsible for downsampling and feature extraction of the input image, and is mainly composed of the WD downsampling module and the DEB convolution block. By stacking multiple convolutional layers, the network can gradually extract more complex features from low-level to high-level. The WD downsampling block is the wavelet downsampling block. It was originally proposed to solve the problem of spatial information loss in semantic segmentation. Considering that it can significantly improve the ability to restore image details and overall structure in dehazing tasks by effectively extracting multi-scale features and retaining important high-frequency information. Therefore, it is applied to our improved dehazing model. The WD module structure is as follows: Figure 3 As shown, it first takes the input feature map X i After discrete wavelet transform, a low-frequency component LL and three high-frequency components are obtained: horizontal high-frequency component HL, vertical high-frequency component LH, and diagonal high-frequency component HH. Then the low-frequency component and the three high-frequency components are concatenated along the channel dimension to form a new feature map. The new feature map is subjected to 1×1 convolution, batch normalization, activation function Relu, and 3×3 convolution to obtain the downsampled feature map X. o The main purpose of this part of the operation after the wavelet transform is to adjust the number of channels of the feature map to align with the subsequent layers and filter out as much redundant information as possible. The calculation formulas for the four components obtained by the wavelet transform are:

[0038] y LL (x,y)=∑ m ∑ n I(m,n)h(m-2x)h(n-2y) (3)

[0039] y HL (x,y)=∑ m ∑ n I(m,n)h(m-2x)g(n-2y) (4)

[0040] y LH (x,y)=∑ m ∑ n I(m,n)g(m-2x)h(n-2y) (5)

[0041] y HH (x,y)=∑ m ∑ n I(m,n)g(m-2x)g(n-2y) (6)

[0042] Where h and g are low-pass and high-pass filters respectively, and wavelet basis is selected for filtering. m and n represent the coordinates of the original image, x and y represent the coordinates of the transformed image, and I is the input feature map. The final output can be expressed as:

[0043] X o =C 3×3 (r(BN(C 1×1 ([LL,HL,LH,HH])))) (7)

[0044] BN is the batch normalization layer, R is the activation function Relu. LL, HL, LH, HH are the four components obtained by wavelet transform, and C is the convolution.

[0045] Step 4: The feature conversion part is located in the deeper level of the model, combining the global features obtained by downsampling. At this stage, the model already has enough contextual information and fog distribution information to effectively refine the defogging task. The feature transformation part of the model is mainly composed of the DEAB convolution block and the SA-Fusion feature fusion module. Compared with the DEB convolution block in other parts, the DEAB convolution block integrates a variety of complex attention mechanisms, which enables it to further calibrate the features, gradually suppress the feature areas with greater fog influence, and highlight the features of the clean image. The SA-Fusion feature fusion module selects the attention feature fusion module, and its structure is as follows: Figure 4 As shown. The core idea of ​​the module is to adaptively select the most useful features from multiple feature maps for fusion through an attention mechanism. This module concatenates, reshapes, and sums all feature maps, calculates the global attention weight, and performs weighted summation on the features to ultimately obtain a more representative and informative output. This mechanism can help the model fuse features from low-level to high-level when dealing with complex dehazing tasks, ensuring that features from different layers can be reasonably weighted, thereby better restoring clear images. And because it does not rely on multiple other attention mechanisms, SA-Fusion provides a simpler and more efficient fusion method. The formula for calculating weights can be expressed as:

[0046] {a1,a2}=Split(Softmax(F mlp (AAP(F low +F high )))) (8)

[0047] where F low is a low-level feature, F high is a high-level feature, AAP is adaptive average pooling, F mlp is the multilayer perceptron in the middle of the fusion module. Multiply the obtained weights with the corresponding feature maps and then add the two results to get the final output:

[0048] F o =F low ×a1+F high ×a2 (9)

[0049] Step 5: The decoding part of the model consists of the DEB convolution block, the SA-Fusion feature fusion module and the BTU upsampling module. This part gradually begins to restore the details of the image, removes the influence of fog and restores it to a nearly fog-free state. The present invention combines bilinear interpolation and transposed convolution as an upsampling module BTU to replace the original transposed convolution upsampling. The BTU structure is as follows: Figure 5 As shown in the figure. By first interpolating and upsampling, the artifacts caused by transposed convolution can be reduced. Performing smooth interpolation before convolution can help improve the overall quality of the feature map and make the details richer. The upsampling module is used to gradually enlarge the downsampled features to the size of the original input image. After upsampling, the DEB convolution block further processes the enlarged features to ensure that key information is not lost during the enlargement process. In the decoding part, the feature fusion module SA-Fusion ensures that the model can comprehensively utilize global context information and local details by fusing features at different levels. This is critical for dehazing, because dehazing requires combining global and local information to reconstruct a real image. Finally, we add a global residual at the end of the model, which can help the model retain more original information while removing fog, reduce the difficulty of training, prevent over-processing, and improve the convergence speed and robustness of the model.

[0050] The loss function of the network adopts contrast loss. By combining the deep features of VGG19 and L1 loss, an effective contrast loss mechanism is formed, which can improve the quality of model-generated images during training. It is designed to optimize image generation by utilizing the relationship between features, so that the model can better understand and reproduce the characteristics of real images. First, the loss function receives three images as input, namely: image a generated by the dehazing model, fog-free image p, and foggy image n. Through the VGG19 network, the features of a, p, and n are extracted respectively. The L1 distance d_ap between the features of a and p is calculated to indicate the similarity between the dehazed image and the real fog-free image. The L1 distance d_an between the features of a and n is calculated to indicate the similarity between the dehazed image and the original foggy image. The calculation of L1 distance is:

[0051]

[0052] Where N is the number of samples, y i is the true value, is the predicted value. The contrast loss calculation formula is:

[0053]

[0054] This ratio measures the similarity between the generated image and the clear image and its similarity to the foggy image. The goal of the model is to minimize this loss so that the generated defogging image a is as close to the fog-free image p as possible and far away from the foggy image n, thereby improving the defogging effect.

[0055] Step 6: Input the training image data in the training set into the created network model and use GPU for training. The number of training iterations is 40,000 and the initial learning rate is 1e -4 , the cosine annealing strategy is used to adjust the learning rate from the initial value to 1e -6 , using the Adam optimizer and a batch-size of 8. During model training, the model parameters with the best accuracy will be saved and named best.pk.

[0056] The test image is used as input to obtain the dehazed image, and the structural similarity SSIM (StructuralSimilarity Index) and peak signal-to-noise ratio PSNR (Peak Signal-to-Noise Ratio) are used as evaluation indicators to evaluate the dehazed effect. Peak signal-to-noise ratio is a commonly used image quality evaluation indicator, which measures the reconstruction quality of the image based on MSE (mean square error). The higher the PSNR, the better the image quality. Its formula can be expressed as:

[0057]

[0058] Where MAX is the maximum value that the image pixel can take, MSE is the mean square error between the original image and the reconstructed image, m and n are the number of rows and columns of the image respectively. I(i,j) is the pixel value at position (i,j) of the original image, and K(i,j) is the pixel value at position (i,j) of the comparison image.

[0059] Structural similarity is a more advanced image quality assessment indicator that not only considers the difference in pixel values, but also the structural information of the image. SSIM is between 0 and 1, and the closer the value is to 1, the better the image quality. Its formula can be expressed as:

[0060]

[0061] Among them, μ x and μ y are the means of images x and y, respectively. and is the variance, σ xy is the covariance, C1 and C2 are small constants used to stabilize the division operation: C1 = (k1L) 2 , C2=(k2L) 2, L is the dynamic range of pixel values, the default values ​​are k1=0.01, k2=0.03.

[0062] The experimental results on the O-HAZE dataset are shown in Table 1. Compared with DEA-Net, the method of the present invention is superior to DEA-Net in terms of peak signal-to-noise ratio and structural similarity. At the same time, the overall effect of the present invention is better than several existing algorithms. Figure 6 As shown, the present invention not only effectively removes the haze in the image, but also can restore the detailed features of the image. The restoration effect of image brightness and contrast is better than DEA-Net, and no color distortion occurs.

[0063] Table 1

[0064] Model PSNR(dB) SSIM AOD-Net 19.59 0.679 DehazeNet 16.21 0.666 GridDehazeNet 18.92 0.672 GFN 22.58 0.737 DEA-Net 22.76 0.710 Ours 24.25 0.732

[0065] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the image dehazing method based on the improved DEA-Net is implemented.

[0066] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the image defogging methods based on the improved DEA-Net.

Claims

1. An image dehazing method based on improved DEA-Net, characterized in that: The following steps are involved: (1) Obtain foggy image data and corresponding clear images from open source websites as training and test sets; (2) Cropping and preprocessing the training set images; (3) Build a deep learning model, which includes an encoding part, a feature transformation part, and a decoding part; (4) Input the training image data in the training set into the created network model and use the GPU for training to obtain the model parameters with the best accuracy; (5) Take the test image as input and obtain the dehazed image.

2. The image defogging method based on improved DEA-Net according to claim 1, characterized in that: In step (2), the training set is cropped and the crop size is set to 1024 × 1024. Then, the training set images are preprocessed before being input into the deep learning model for training: randomly cropped to 256 × 256 size and randomly rotated 90 degrees.

3. The image defogging method based on improved DEA-Net according to claim 1, characterized in that: In step (3), the encoding part of the model is mainly responsible for downsampling and feature extraction of the input image, and is composed of a downsampling module and a convolution block; wherein the convolution block for feature extraction utilizes the detail enhancement block DEB in DEA-Net, and the downsampling block utilizes the WD downsampling block; wherein the WD downsampling block is a wavelet downsampling block.

4. The image defogging method based on improved DEA-Net according to claim 3, characterized in that: In step (3), the wavelet downsampling block is as follows: First, let the input feature map X i After discrete wavelet transform, a low-frequency component LL and three high-frequency components are obtained: horizontal high-frequency component HL, vertical high-frequency component LH, and diagonal high-frequency component HH; then the low-frequency component and the three high-frequency components are spliced ​​along the channel dimension to form a new feature map; the new feature map is subjected to 1×1 convolution, batch normalization, activation function Relu, and 3×3 convolution to obtain the downsampled feature map X o .

5. The image defogging method based on improved DEA-Net according to claim 1, characterized in that: In step (3), the feature transformation part of the model is constructed. The feature transformation part is located at a deeper level of the model and is used to enhance feature expression and gradually remove the interference of fog on the image. The feature transformation part is composed of a convolution block and a feature fusion module. The convolution block uses the attention block DEAB in DEA-Net. The feature fusion module adopts SA-Fusion, which adaptively selects the most useful features from multiple feature maps for fusion through an attention mechanism.

6. The image defogging method based on improved DEA-Net according to claim 5, characterized in that: In step (3), the SA-Fusion formula is as follows: {a1,a2}=Split(Softmax(F mlp (AAP(F low +F high , Among them, F low is a low-level feature, F high is a high-level feature, AAP is adaptive average pooling, F mlp is the multilayer perceptron in the middle of the fusion module. Multiply the obtained weights with the corresponding feature maps and then add the two results to get the final output: F o =F low ×a1+F high ×a2 7. The image defogging method based on improved DEA-Net according to claim 1, characterized in that: In step (3), the decoding part of the model consists of a convolution block, a feature fusion module and an upsampling module; the convolution block adopts the detail enhancement convolution block DEB, and the upsampling block adopts the BTU upsampling module that integrates bilinear interpolation and transposed convolution. In feature fusion, the features after the downsampling operation and the corresponding features before the upsampling operation are fused through the SA-Fusion fusion module; a global residual is added at the end of the network to retain the original input information.

8. The image defogging method based on improved DEA-Net according to claim 1, characterized in that: Step (4) is as follows: the number of training iterations is 40,000, and the initial learning rate is 1e -4 , the cosine annealing strategy is used to adjust the learning rate from the initial value to 1e -6 , Adam optimizer is used with a batch-size of 8; the test image is taken as input to obtain the dehazed image, and the structural similarity and peak signal-to-noise ratio are used as evaluation indicators to evaluate the dehazing effect.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, an image defogging method based on improved DEA-Net according to any one of claims 1 to 8 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, an image defogging method based on improved DEA-Net is implemented according to any one of claims 1 to 8.

Images