Attention-Optimized Deep Encoder-Decoder Dehazing Generative Adversarial Network

Abstract

Aiming at the problems that existing dehazing algorithms are difficult to handle unevenly distributed fog and that deep convolutional dehazing networks overly rely on large-scale datasets, the present invention proposes an attention-optimized deep encoder-decoder dehazing generative adversarial network. The network adopts an encoder with a four-layer downsampling structure to fully extract the semantic information lost due to fog in order to restore a clear image. At the same time, in the decoder network, an attention mechanism is introduced to adaptively assign weights to different pixels and channels, thereby dealing with unevenly distributed fog. Finally, the framework of the generative adversarial network enables the model to achieve better training results on small-sample datasets. Experimental results show that the technical solution of the present invention can not only effectively remove unevenly distributed fog in real-scene images, but also restore clear images for real-scene datasets with fewer training samples, and the evaluation indexes are better than other widely used comparison algorithms.

Description

Technical Field

[0001] This technical solution belongs to the field of image processing, and specifically, it is a deep encoding-decoding dehazing generative adversarial network with attention optimization applied to image processing. Background Art

[0002] The scattering phenomenon that occurs when light passes through suspended particulate matters such as fog and haze degrades the images captured by imaging sensors, and thus a large amount of effective information is lost, directly affecting the execution effect of downstream high-level vision tasks. The purpose of image dehazing is to eliminate the influence of the atmospheric environment on image quality, increase the visibility of the image, and provide support for downstream semantic tasks (such as image classification, object detection, etc.).

[0003] Image dehazing is mainly divided into traditional dehazing and deep learning dehazing algorithms. Most traditional dehazing algorithms use prior knowledge to estimate parameters to restore the original image [1, 2] and are based on the atmospheric scattering model [3]. Among them, the dark channel prior dehazing algorithm (DCP) [1] is recognized by researchers as having good dehazing performance and being the most widely used. This algorithm realizes image dehazing by calculating the transmission rate and the estimated atmospheric light, without the need for additional parameters, but it is almost ineffective in images with high brightness. The color attenuation prior algorithm (CAP) [2] finds the foggy areas by comparing the relationship between image saturation and brightness change. Compared with the DCP algorithm, the CAP algorithm improves the method of calculating the image depth of field and finds the most suitable parameter values through deep learning algorithms, so the effect is better than the DCP algorithm. Other prior dehazing algorithms [4 - 6] also focus on the differences in features between foggy images and fog-free images and thus determine the foggy areas. Although traditional algorithms have low complexity, they also have the following inevitable defects: First, due to the superposition of errors during parameter estimation and the inaccuracy of the atmospheric scattering model, the generated dehazed images cannot fully restore the reference images. Second, the performance of traditional algorithms is largely limited by the accuracy of fog area detection. When dehazing, the original low-frequency information in the fog-free areas may also be removed. It can be seen that although traditional dehazing algorithms have made great progress, they still have the problem of insufficient robustness when facing real-world scenarios, and it is difficult to apply them in real-world scenarios.

[0004] With the wide application of deep learning in the field of image processing, the dehazing model based on Convolutional Neural Network (CNN for short) shows better performance than traditional algorithms. Some CNN-based dehazing algorithms [7-9] learn key parameters such as the transmission map, atmospheric light, and depth of field between hazy images and haze-free images by training on indoor or outdoor real images and corresponding hazy images generated by computer simulation, and then calculate the predicted dehazed image according to the atmospheric scattering model. Among them, DehazeNet [7], as the earliest deep learning dehazing model, uses modules such as multi-scale feature extraction and non-linear regression to estimate the transmission map and then substitutes it into the atmospheric scattering model to restore the haze-free image. AOD-Net [8] is divided into two serial modules. The K estimation module is used to estimate the depth information and relative level of the fog, and the image generation module generates pictures. Compared with DehazeNet, AOD-Net has better results in processing natural images and less loss to the original image. DCPDN [9] uses two parallel sub-modules. The pyramid dense connection encoding and decoding module is used to estimate the transmission map, and the U-shaped feature extraction module is used to estimate the atmospheric light. Finally, these two parameters are substituted into the atmospheric scattering model to calculate the dehazed image. Inspired by other low-level image task algorithms, new dehazing models

[10]

[11] choose a direct end-to-end network to avoid the atmospheric scattering model. Such a design can not only avoid the error accumulation caused by the atmospheric scattering model but also be more conducive to supporting downstream high-level semantic tasks.

[0005] Although the deep learning-based image dehazing algorithm has achieved good results, it still faces the following problems:

[0006] First, the performance of the fully end-to-end network overly depends on the training results on large-scale datasets. Most of the dehazing datasets used are synthesized by computer simulation, which has a gap with the real scene. And it is difficult to obtain large-scale real-scene dehazing datasets, which limits the performance of the model.

[0007] Second, most of the existing methods process the pixel points on the pictures without discrimination and cannot handle the unevenly distributed fog in the real scene. Summary of the Invention

[0008] To solve the above two problems, the present invention proposes an attention-optimized deep encoding and decoding dehazing generative adversarial network, which is characterized by including a generator network and a discriminator network. The steps include: 1) The generator network receives the input hazy image and generates a dehazed image; 2) The discriminator network receives the dehazed image output by the generator network and the corresponding real haze-free image and makes a judgment; 3) The discriminator network supervises the training process of the generator network.

[0009] In the generator network: First, the input foggy image is preprocessed to obtain the original feature map. Then, an encoder module with four layers of downsampling is applied to obtain the encoded feature map. The feature map is processed through a residual module and sent into the decoder module. The decoder module is also divided into four layers to restore the size of the feature map to its original size; when reconstructing the feature map, an attention module is introduced; for the encoder and decoder of the same size, the feature map obtained by encoding the encoder is spliced and fused into the input feature map of the decoder. Finally, the feature map restored to the original size is sent into the post-processing module for channel reduction and added to the input foggy image to finally obtain the de-fogged image of the input.

[0010] The technical features of this generative adversarial network mainly include:

[0011] (1) The four-layer downsampling encoder with dense connections can fully extract the deep semantic information lost due to being obscured by thick fog, while the residual connection ensures that shallow information such as contours, contrast, and texture does not disappear when being transmitted to the deep layer;

[0012] (2) The attention module added to the decoder layer can adaptively find the thick fog and thin fog regions and channels in the picture and assign different weights to them, so that the network has the ability to cope with unevenly distributed fog;

[0013] (3) Adopting the framework of the generative adversarial network enables the system to have stronger robustness even when trained on a small sample dataset.

[0014] Through the framework of the generative adversarial network in the present invention, the model can achieve good training results in a small sample dataset. Among them, the generator network adopts a deep encoding and decoding structure. The encoder adopts four layers of deep downsampling to ensure full extraction of the text information of the image in order to recover the information loss caused by thick fog. An attention mechanism is introduced in the decoder, which can adaptively assign attention weights to different pixels and channels to cope with the uneven distribution of fog in the real scene. The experimental results on the small sample real datasets I-HAZY and O-HAZY show that compared with the widely used de-fogging algorithms and models, the network model of the present invention has excellent performance in both objective indicators and visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the generator network;

[0016] Figure 2 It is a schematic diagram of the decoder structure;

[0017] Figure 3 It is a schematic diagram of the discriminator network;

[0018] Figure 4It is a schematic diagram for comparing upsampling methods;

[0019] Figure 5 It is a schematic diagram for comparing the LeakyReLU and ReLU activation functions;

[0020] Figure 6 It is a schematic diagram for comparing defogging effects. Specific implementation manners

[0021] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners:

[0022] 1 Overview

[0023] The attention-optimized deep encoder-decoder defogging generative adversarial network of the present invention is a defogging generative adversarial network based on an encoder-decoder architecture. The generative adversarial network of the present invention adopts an encoder with a four-layer downsampling structure to fully extract the semantic information lost due to fog in order to restore a clear image. At the same time, in the decoder network, an attention mechanism is introduced to adaptively assign weights to different pixels and channels, thereby dealing with unevenly distributed fog. Finally, the framework of the generative adversarial network enables the model to achieve better training effects on a small-sample dataset.

[0024] Experimental results show that the defogging network of the present invention can not only effectively remove unevenly distributed fog in real-scene images, but also restore clear images for real-scene datasets with fewer training samples, and the evaluation indexes are better than other comparative algorithms widely adopted.

[0025] 2 Algorithm implementation

[0026] 2.1 Introduction to related technologies

[0027] 2.1.1 Generative adversarial network

[0028] The generative adversarial network

[12] (Generative Adversarial Network, abbreviated as GAN) effectively learns the distribution of training images to synthesize real images through the min-max optimization principle of game theory, which is more similar to humans learning knowledge in an unsupervised situation. The generative adversarial network can improve the problems that traditional convolutional neural networks highly depend on the number of samples and the quality of labels.

[0029] 2.1.2 Encoder-decoder structure

[0030] The application of the encoding and decoding structure in the field of deep learning was first in the sequence-to-sequence (Seq-to-Seq) model

[13] in the field of natural language processing. The UNet network framework of the encoding and decoding structure can pay more attention to the semantic information of images in the field of image processing, and reconstruct feature maps at different scales to match the original size. This feature is particularly important for end-to-end tasks. At the same time, UNet adds long skip connections to the encoding and decoding layers of the same scale, which ensures that the shallow semantic information will not be forgotten when reconstructing the image and prevents the vanishing gradient.

[0031] 2.1.3 Fully End-to-End Network

[0032] Although the atmospheric scattering model explains the generation of foggy images, using the atmospheric scattering model as a benchmark for dehazing algorithms will lead to the accumulation of errors. This is because the atmospheric scattering model is only a physical simulation model for foggy environments. There are non-negligible errors in the calculation of the transfer map, atmospheric light components, background radiation, and depth of field compared with the real scene, which limits the performance of dehazing algorithms. The learning ability of neural networks enables us to abandon the atmospheric scattering model and directly learn the distribution of fog on the image, thus obtaining dehazed images in a fully end-to-end manner, which is also beneficial for industrial deployment.

[0033] 2.2 Dehazing Generative Adversarial Network Model

[0034] 2.2.1 Overall Model Framework

[0035] The model adopts the framework of the generative adversarial network, which is divided into a generator network and a discriminator network. The generator network receives the input foggy image and generates a dehazed image. The discriminator network receives the dehazed image output by the generator network and the corresponding real fog-free image, makes a judgment, and at the same time supervises the training process of the generator network. In this model, as Figure 1 shown, the generator network adopts a four-layer mirror-like deep encoding and decoding structure to fully extract the text information of the image. At the same time, the long skip connection strategy is also applied to ensure that the shallow information is not lost during the transmission process and prevent the vanishing gradient. As Figure 2 shown, a specially designed attention module is added to the decoder module to deal with unevenly distributed fog. As Figure 3 shown, the discriminator network is a common binary classification network. Through the framework of the generative adversarial network, the model can reduce the requirements for large-scale training datasets and broaden the application scenarios of the model.

[0036] 2.2.2 Four-Layer Downsampling Encoder with Dense Residual Connections

[0037] The present invention designs a four-layer dense residual connection to replace the original encoder in the UNet model. This encoder utilizes DenseNet-121

[15] pre-trained on the ImageNet

[14] dataset as a feature extractor. The remarkable feature of DenseNet is the use of a densely connected residual network. Compared with simply stacking convolutional layers directly or using ResNet to extract features, while fully extracting image text information, DenseNet improves the feature utilization rate without introducing additional parameters, which is beneficial for restoring the image information lost due to high-concentration fog. In addition, it establishes sufficient skip connections between different layers, fusing the features extracted by the previous layer onto the current layer to ensure the accuracy of visual tasks.

[0038] The encoder structure of the present invention is as Figure 1 shown. The present invention first introduces a preprocessing process, which includes a Conv layer, a BatchNorm

[16] layer, and a ReLU

[17] layer. The original features obtained after preprocessing are then fed into the encoder. The encoder consists of four consecutive groups of pre-trained DenseBlocks and TransBlocks. Both the DenseBlock and TransBlock layers are standard DensNet-121. After being pre-trained on the ImageNet dataset, pre-trained weights are used to replace randomly initialized weights to accelerate the model convergence speed. Each TransBlock layer contains a max pooling layer, which reduces the features by 1 / 2, and finally obtains a feature map with an original size of 1 / 16.

[0039] 2.2.3 Attention-Enhanced Decoder

[0040] Since the encoder reduces the feature size to 1 / 16 of the original size, a mirror-image four-layer upsampling module is required to restore the feature size. Similar to the encoder, the present invention uses four consecutive groups of DenseBlocks and TransBlocks. However, the DenseBlocks and TransBlocks in the decoder are different from those in the encoder and are specially designed and simplified, aiming to process the sampled feature maps and restore the feature map size to the size of the original image. The structures of the simplified DenseBlock, TransBlock, and attention module are as Figure 3As shown, each DenseBlock of the decoder includes two groups of ReLU layers and Conv layers, and a residual connection is made between the input feature map of the DenseBlock and the processed feature map. To restore the size of the feature map to the original size, an upsampling operator must be included in each TransBlock. Compared with direct bilinear interpolation for upsampling, the present invention uses a learnable upsampling module PixelShuffle

[18] , as Figure 4 shown, to avoid the artifacts generated by interpolation during upsampling, which is more suitable for end-to-end image tasks.

[0041] Common defogging methods assign uniform weights to all pixels and channels, which does not conform to real-world experience. Since fog is unevenly distributed in real-world scenes, setting the same weight for all pixel values will result in insufficient defogging in areas with higher fog concentration, affecting the final image restoration quality. Moreover, for all channels of the feature map, their sensitivities to fog concentration are also different. Therefore, when reconstructing the feature map, introducing an attention module helps the network better learn the fog concentration distribution and achieve a more ideal defogging effect. Specifically, behind the PixelShuffle upsampling module in each TransBlock, an attention module is added, and its structure is as Figure 2 shown. This attention module is divided into a channel-domain attention branch and a spatial-domain attention branch. In the channel-domain attention branch, the present invention first takes the average value of all pixels in each channel of the feature map processed by the DenseBlock in the decoder as the original channel attention feature vector. For a feature map with a dimension of c*h*w, a feature vector with a size of c*1 can be obtained. To enable the feature vector to learn the sensitivities of different channels, the present invention then sequentially performs Conv, ReLU, Conv, and Sigmoid operations on it. In this way, through training, this feature vector can learn the attention of different channels. Finally, multiplying this attention vector by the original feature map yields the channel attention feature map. The calculation of spatial-domain attention is similar to that of the channel domain. For the obtained channel attention feature map, the present invention first performs average pooling and max pooling operations on it along the channel direction to obtain two original spatial attentions of 1*h*w. The pixel at each point on this feature map is respectively the average value and the maximum value of all pixels at this position on different channels. Immediately afterwards, Conv and Sigmoid operations are performed on these two feature maps to obtain the learned spatial-domain attention. Finally, multiplying the spatial-domain attention by the feature map mixed with the channel-domain attention enables the feature map to learn the uneven distribution of fog in both channels and space.

[0042] 2.2.4 Generator Network and Discriminator Network

[0043] The model of the present invention adopts the framework of a generative adversarial network. Its generator network is different from the direct encoder-decoder structure. Instead, it refers to the long skip connection strategy of UNet, and splices and fuses the feature maps obtained by encoding the encoder of the same size into the input feature maps of the decoder. The adoption of long skip connections can connect the information of the input image, help to restore the information loss caused by downsampling, and avoid additional parameters at the same time.

[0044] The input of the generator network is a foggy image, which first undergoes a preprocessing process to obtain the original feature map. Then, an encoder module with four layers of downsampling is applied to obtain the encoded feature map, whose size is 1 / 16 of the original image size. Immediately afterwards, the feature map is processed by a residual module and sent into the decoder module. The decoder module is also divided into four layers to restore the feature map size to the original size. Each layer of the decoder contains a designed attention module. Finally, the feature restored to the original size is sent into a post-processing module containing a Padding layer, a Conv layer, and a Tanh layer to be restored to three channels, and added to the input foggy image to finally obtain the de-fogged image of the input.

[0045] The discriminator network is a conventional binary classification network, and its input is the de-fogged image generated by the generator. The output of the discriminator network is true or false, which evaluates the quality of the de-fogged image output by the generator network and guides the training of the generator network to adjust the parameters accordingly. In the discriminator, the present invention uses a series of combinations of LeakyReLU layer

[20] , Conv layer, and BatchNorm layer. The LeakyReLU function, as Figure 5 shown, in the process of gradient backpropagation, compared with the conventional ReLU function, a hyperparameter is added to calculate the gradient when the input of the activation function is less than 0, which solves the problem of neuron death caused by the appearance of negative samples. The BatchNorm layer can centralize and normalize each batch. This operation can avoid the continuous increase of parameters when the parameter changes too much due to different data distributions. Of course, it can also avoid gradient explosion, and speed up the convergence rate while using a more accurate learning rate. These two layers have good applications in the classification network.

[0046] 2.2.5 Loss Function

[0047] In order to comprehensively consider various aspects of the generated de-fogged image and better guide the model to complete training, the present invention uses an integrated loss function L, which is composed of a reconstruction loss function L r , a perceptual loss function L p , and an adversarial loss function L a , and its definition formula is shown in Formula (1):

[0048] L = αL r + βLp +γL a (1)

[0049] Among them, α, β, and γ are the weight coefficients of the corresponding loss functions.

[0050] The reconstruction loss function L r has the calculation formula shown in Equation (2):

[0051]

[0052] Among them, L1 represents the mean absolute error between the dehazed image generated by the generator and the real haze-free comparison image, and I i is the input blurred image; G(·) represents the operation of the generator's dehazing network, G(I i ) and J i represent the dehazed image generated by the generator and the real haze-free comparison image respectively;

[0053] The reconstruction loss measures the mean absolute error between the generated image and the reference image, which is an objective indicator. Simply using the reconstruction loss to guide the model training will lead to a situation where the objective result does not match the subjective visual perception. The perceptual loss is proposed to solve this problem. The perceptual loss function L p has the calculation formula shown in Equation (3):

[0054]

[0055] Among them, L2 represents the mean variance error, vgg(·) represents the pre-trained VGG16 network; G(·) represents the operation of the generator's dehazing network, G(I i ) and J i represent the dehazed image generated by the generator and the real haze-free comparison image respectively; The VGG16 network is a network trained on ImageNet, which can extract the edges, colors, brightness, textures, and even more profound and imperceptible semantic features of the image. Using VGG16 to simulate the difference in human eye perception of images has also been widely applied to tasks such as super-resolution and style transfer.

[0056] The adversarial loss function L a is also integrated into the loss function to reflect the guiding training role of the discriminator network, and its definition formula is shown in Equation (4):

[0057]

[0058] Among them, D(G(I i )) is the probability that the discriminator network judges that the dehazed image G(I i ) generated by the generator is true.

[0059] 3 Experimental Procedures and Result Analysis

[0060] 3.1 Datasets and Performance Metrics

[0061] For the consideration of application effects in real scenarios, the present invention selects the small-sample real datasets I-HAZY

[23] and O-HAZY

[24] . The I-HAZY and O-HAZY datasets are proposed to solve the problem that the current deep learning-based haze removal methods rely too much on large-sample synthetic datasets. Compared with the most commonly used haze removal datasets, the I-HAZY and O-HAZY datasets are more challenging for the performance of the model.

[0062] The I-HAZY dataset includes 30 pairs of indoor real hazy images and corresponding haze-free images, of which 25 pairs are used for training and 5 pairs are used for testing. The 0-HAZY dataset includes 45 pairs of outdoor real hazy images and corresponding haze-free images, of which 40 pairs are used for training and 5 pairs are used for testing. Among them, the real hazy images are all generated by a professional haze generator and taken under the same lighting parameters as the corresponding haze-free images, which is closer to actual applications.

[0063] The present invention uses two objective performance metrics and one subjective metric commonly used in the research on image restoration in domestic and foreign literatures to evaluate the performance of the model proposed by the present invention and conduct a comparative analysis with other haze removal methods. The objective metrics are Peak Signal to Noise Ratio (PSNR for short) and Structural Similarity index (SSIM for short), and the subjective metric is Learned Perceptual Image Patch Similarity (LPIPS for short)

[25] .

[0064] PSNR is the most commonly used objective evaluation metric for image quality, which is used to measure the pixel-level error between an image and its corresponding reference image. PSNR is an error-sensitive image quality evaluation metric, and its definition formula is shown in Equation (4):

[0065]

[0066] In the formula, n represents the bit width of the pixel, and MSE is the mean absolute error between the haze-removed image and its corresponding reference image.

[0067] SSIM considers the errors between the haze-removed image and its corresponding reference image from three aspects: brightness, contrast, and structure. Compared with PSNR, SSIM is more comprehensive and conforms to people's intuitive feelings, and its definition formulas are shown in Equation (5) and Equation (6):

[0068]

[0069] SSIM = l(x, y) * c(x, y) * s(x, y) (7)

[0070] where μ represents the mean value, and σ x represents the variance, and σ xy represents the covariance. l, c, and s represent luminance, contrast, and structure respectively, and c1, c2, and c3 are all constants.

[0071] The LPIPS metric measures the similarity of high-order image structures to replace the actually impossible-to-constitute distance metric, which is more in line with human perception of similarity. In actual use, LPIPS uses a pre-trained deep network on the ImageNet dataset to extract the deep features of the image and the reference image. The lower the LPIPS value, the higher the feature similarity between the generated image and the corresponding reference image, and the more similar the subjective perception is.

[0072] 3.2 Experimental Parameters and Environment Settings

[0073] The program of the present invention is written using the Pytorch framework, and the program code website is https: / / kirqwer6666 / dehazeGAN, and it is trained on a computer configured with an Intel i9 9900K CPU and an NVIDIA Geforce RTX 2080Ti GPU. The initial learning rate is set to 0.0001, and ADAM

[26] is used as the learning rate optimization strategy, and StepLR is used to adjust the base learning rate stage by stage. The pictures are cropped into a fixed size and randomly rotated by 0°, 90°, 180°, or 270° to prevent overfitting. To further improve the robustness of the generative adversarial network, the present invention sets up a sample pool. When the generator receives 50 samples, the subsequent reference real images will have a 50% probability of matching the correct hazy image and a 50% probability of matching another image in the random sample pool. The purpose of this is to prevent the discriminator network from stopping training and to supervise the training of the generator network on a small sample dataset. During the training process, num of threads is set to 8, batch size is set to 1, epoch is set to 5000, learning rate step is set to 1000, and learning rate decay is set to 0.5. The last two parameters mean that every 1000 epochs of training, the base learning rate will be decayed to half of the initial learning rate.

[0074] 3.3 Experimental Results and Analysis

[0075] The model of the present invention is compared with DCP[1], CAP[2], DehazeNet[7], AOD-Net[8] and DCPDN[9]. The visual comparison results are as follows: Figure 6 The experimental results of quantitative analysis are shown in Table 1 and Table 2 respectively.

[0076] Compare the effects of various algorithms and models on I-HAZY and O-HAZY datasets. Figure 6 As can be seen, the Dark Channel Prior (DCP) algorithm completely fails on both indoor and outdoor datasets. This is because the high brightness of the images causes the images to not satisfy the Dark Channel Prior law. The Color Attenuation Prior (CAP) algorithm offers some performance improvements over the Dark Channel Prior algorithm, but it still only handles scenes with relatively thin fog. It remains ineffective for scenes with densely distributed fog in outdoor images or fog in areas with a deep depth of field. The results of DehazeNet and AOD-Net are relatively similar. Whether for indoor or outdoor images, they can remove surface fog, making the haziness caused by fog less. However, due to the lack of extraction and application of deep information, they cannot achieve better results. Although the DCPDN model has a relatively good dehazing effect, it suffers from color distortion, which is more obvious in outdoor images. Compared with these models, the model of the present invention achieves a more thorough dehazing effect, whether for indoor or outdoor images, dense fog or uneven fog distribution, with less color restoration distortion, more detail preservation, and no obvious haziness caused by residual fog.

[0077] Table 1 PSNR and SSIM comparison of dehazing effect

[0078]

[0079] Table 2 Comparison of LPIPS dehazing effect

[0080]

[0081] From the quantitative analysis of objective indicators in Table 1, we can see that for the I-HAZY and O-HAZY datasets, the PSRN of our model is 6.45 and 5.94 higher than the optimal values of the five compared models, respectively. For the O-HAZY dataset, the SSIM of our model is 0.103 higher than the optimal value of the five compared models. From the quantitative analysis of subjective indicators in Table 2, we can see that for the O-HAZY dataset, the LPIPS of our model is 0.108 lower than the optimal value of the five compared models. Therefore, the model proposed in this paper significantly outperforms the other five models from both objective and subjective perspectives. This advantage is even more pronounced for outdoor foggy images with uneven fog concentration.

[0082] 4 Conclusion

[0083] Attention-Optimized Deep Encoder-Decoder Dehazing Generative Adversarial Network of the present invention. Through the framework of the generative adversarial network, the model can achieve good training results in small-sample datasets. Among them, the generator network adopts a deep encoder-decoder structure. The encoder adopts four-layer deep downsampling to ensure sufficient extraction of the text information of the image, in order to recover the information loss caused by thick fog. An attention mechanism is introduced in the decoder, which can adaptively assign attention weights to different pixels and channels, so as to cope with the uneven distribution of fog in real scenes. Experimental results on small-sample real datasets I-HAZY and O-HAZY show that compared with widely used dehazing algorithms and models, our model has excellent performance in both objective metrics and visual perception.

[0084] References:

[0085] [1] He K, Sun J, Tang X. Single image haze removal using dark channel prior[J]. IEEE transactions on pattern analysis and machine intelligence, 2010, 33(12): 2341-2353.

[0086] [2] Zhu Q, Mai J, Shao L. A fast single image haze removal algorithm using color attenuation prior[J]. IEEE transactions on image processing, 2015, 24(11): 3522-3533.

[0087] [3] Freeman, William T., and Edward H. Adelson. "The design and use of steerable filters." IEEE Transactions on Pattern analysis and machine intelligence 13.9(1991): 891-906.

[0088] [4] He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." European conference on computer vision. Springer, Berlin, Heidelberg, 2010.

[0089] [5] Cao, Xumin, Liu, Chunxiao, Zhang, Jindong, Lin, Yuhang, Zhao, Jinwei. A fast image dehazing algorithm based on brightness contrast enhancement and saturation compensation [J]. Journal of Computer-Aided Design & Computer Graphics, 2018, 30(10): 1925 - 1934.

[0090] [6] Shen, Yiyun, Liu, Chunxiao, Zhang, Jindong, Shao, Yaqi, Zhao, Jinwei. An atmospheric light correction and transmittance optimization algorithm for robust image dehazing [J]. Journal of Computer-Aided Design & Computer Graphics, 2017, 29(09): 1604 - 1612.

[0091] [7] Cai, B, Xu, X, Jia, K, et al. Dehazenet: An end-to-end system for single-image haze removal [J]. IEEE Transactions on Image Processing, 2016, 25(11): 5187 - 5198.

[0092] [8] Li, B, Peng, x, Wang, Z, et al. Aod-net: All-in-one dehazing network [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 4770 - 4778.

[0093] [9] Zhang, H, Patel, V M. Densely connected pyramid dehazng network [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3194 - 3203.

[0094]

[10] Zhao Yang, Li Bo. Single Image Dehazing Based on Enhanced Generator Conditional Generative Adversarial Network [J / OL]. Computer Applications: 1-8 [2021-06-20]. http: / / kns.cnki.net / kcms / detail / 51.1307.TP.20210601.0904.004.html.

[0095]

[11] Zhang Jinlong, Yang Yan. End-to-End Dehazing Algorithm Based on Fog Layer Feature Extraction and Enhancement Network [J / OL]. Journal of Measurement Science and Instrumentation: 1-13 [2021-06-20]. http / / kns.cnki.net / kcms / detail / 14.1357.TH.20210507.1534.002.html.

[0096]

[12] Goodfellow, Ian J., et al. "Generative adversarial networks." arXiv preprint arXiv: 1406.2661(2014).

[0097]

[13] Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." arXiv preprint arXiv: 1409.3215(2014).

[0098]

[14] Russakovsky, Olga, et al. "Imagenet large scale visual recognition challenge." International journal of computer vision 115.3(2015): 211-252.

[0099]

[15] Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 4700-4708.

[0100]

[16] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[J]. arXiv preprint arXiv:1502.03167, 2015.

[0101]

[17] Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks[C] / / Proceedings of the fourteenth international conference on artificial intelligence and statistics. 2011:315 - 323.

[0102]

[18] Shi, Wenzhe, et al″Real - time single image and video super - resolution using an efficient sub - pixel convolutional neural network.″Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[0103]

[19] Woo, Sanghyun, et al.″Cbam: Convolutional block attention module.″Proceedings of the European conference on computer vision (ECCV). 2018.

[0104]

[20] He, Kaiming, et al.″Delving deep into rectifiers: Surpasslng human - level performance on imagenet classification.″Proceedings of the IEEE international conference on computer vison. 2015.

[0105]

[21] Johnson J, Alahi A, Fei-Fei L. Perceptual losses for real-time style transfer and super-resolution[C] / / European conference on computer vision. Springer, Cham, 2016: 694-711.

[0106]

[22] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for lage-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).

[0107]

[23] Ancuti C, Ancuti C O, Timofte R, et al. I-HAZE: a dehazing benchmark with real hazy and haze-free indoor images[C] / / International Con-ference on Advanced Concepts for Intelligent vision Systems. Springer, Cham, 2018: 620-631.

[0108]

[24] Ancuti C O, Ancuti C, Timofte R, et al. O-haze: a dehazng benchmark with real hazy and haze-ffee outdoor mages[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018: 754-762.

[0109]

[25] Zhang, Richard, et al "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[0110]

[26] Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization" arXiv preprint arXiv:1412.6980 (2014).

Claims

1. An attention-optimized deep encoder-decoder dehazing generative adversarial network, characterized in that It includes a generator network and a discriminator network, and its steps include: 1) The generator network receives the input foggy image and generates a defogged image; 2) The discriminator network receives the defogged image output by the generator network and the corresponding real fog-free comparison image, and makes a judgment; 3) The discriminator network supervises the training process of the generator network; In step 1) described above, the steps for the generator network to generate a defogged image include: 1.1) The input foggy image is preprocessed to obtain an original feature map; 1.2) An encoder module with four layers of downsampling is used to obtain an encoded feature map; 1.3) The feature map is processed by a residual module and sent into the decoder module; 1.4) The decoder module is also divided into four layers, and finally restores the size of the feature map to the original size; when reconstructing the feature map, an attention module is introduced; 1.5) The feature map restored to the original size is sent into the post-processing module, where channel restoration is performed, and it is added to the input foggy image to finally obtain the defogged image of the input; In step 1.4) described above, the decoder includes four consecutive groups of simplified DenseBlock and TransBlock; in each group of DenseBlock and TransBlock: DenseBlock includes two groups of ReLU activation layers and Conv convolutional layers, and the feature map obtained by the corresponding group of the encoder is connected with the input feature map of the corresponding group of the decoder as a residual connection, so as to fuse the encoder output feature map and the decoder input feature map of the same size layer to generate the feature map of each layer of the decoder; Each TransBlock contains an upsampling module PixelShuffle, which doubles the size of the feature map of each layer of the decoder. Through the upsampling modules of the four TransBlocks, the feature map is finally restored to the original size; In step 1.4) described above, when reconstructing the feature map, an attention module is introduced. The method is: Behind the upsampling module PixelShuffle of each TransBlock, an attention module is added; the attention module is divided into a channel domain attention branch and a spatial domain attention branch; 1.4.1) In the channel domain attention branch: First, the average value of all pixels of each channel of the feature map processed by DenseBlock in the decoder is taken as the original channel attention feature vector; For a feature map of c*h*w dimensions, a feature vector of c*1 size is obtained; Then, Conv, ReLU, Conv, and Sigmoid activation functions are sequentially performed on this feature vector; through training, this feature vector learns the attention of different channels; Finally, this attention vector is multiplied by the original feature map to obtain the channel attention feature map; 1.4.2) In the spatial domain attention branch: Adopt a learning method similar to that of the channel attention feature map to obtain the spatial attention feature map; First, perform average pooling and max pooling operations on the feature map fused with channel attention along the channel direction respectively to obtain two original spatial attention feature maps of 1*h*w; the pixel at each point on the feature map is respectively the average value and the maximum value of all pixels at this point on different channels. Then, perform convolution and Sigmoid activation function operations on these two feature maps to obtain the learned spatial domain attention feature map. 1.4.3) Multiply the spatial domain attention feature map by the feature map mixed with channel domain attention, then the feature map generated by the attention module learns the features of fog that are unevenly distributed in channels and space.

2. The attention-optimized deep encoder-decoder defogging generative adversarial network according to claim 1, characterized in that In step 1.1), the image before entering the encoder is preprocessed: the input foggy image passes through a Conv convolutional layer, a BatchNorm batch normalization layer, and a ReLU activation layer in sequence. Obtain the original features of the preprocessed foggy image.

3. The attention-optimized deep encoder-decoder defogging generative adversarial network according to claim 1, characterized in that In step 1.2), the encoder includes four consecutive groups of pre-trained DenseBlock and TransBlock; each group of DenseBlock and TransBlock layers is a standard DensNet-121; DensNet-121 is pre-trained on the ImageNet dataset and uses pre-trained weights instead of randomly initialized weights. Each TransBlock layer contains a max pooling layer that can reduce the image features to 1 / 2 of the original size. Finally, through the four max pooling layers in the four groups of DenseBlock and TransBlock, a feature map of 1 / 16 of the original size is obtained.

4. The attention-optimized deep codec dehazing generative adversarial network according to claim 1, characterized in that In step 1.5), the post-processing module performs channel restoration: the feature map restored to the original size is fed into the post-processing module, and inside the post-processing module are a Padding padding layer, a Conv layer, and a Tanh activation layer in sequence, and finally restored to three channels.

5. The attention-optimized deep codec defogging generative adversarial network according to claim 1, characterized in that In step 2), the discriminator network is a binary classification network; the input of the discriminator network is the de-fogged image generated by the generator network and the corresponding real fog-free comparison image; the output of the discriminator network is true or false, evaluating the quality of the de-fogged image output by the generator network and guiding the training of the generator network to adjust the parameters accordingly. In the discriminator network, a combination of LeakyReLU layer, Conv convolutional layer, and BatchNorm layer is used.

6. The attention-optimized deep codec dehazing generative adversarial network according to claim 1, characterized in that In step 3), the loss function L is an integrated loss function, which is composed of a reconstruction loss function L r , a perceptual loss function L p , and an adversarial loss function L a . Its definition formula is shown in formula (1): L = αL r + βL p + γL a (1) Among them, α, β, γ are the weight coefficients of the corresponding loss functions. Reconstruction loss function L r The calculation formula is shown in Equation (2) as follows: Among them, L1 represents the mean absolute error between the dehazed image generated by the generator and the real haze-free comparison image. Ii is the input blurred and hazy image of the generator network. G(·) represents the operation of the generator dehazing network, G(I i ) and J i represent the dehazed image generated by the generator and the real haze-free comparison image respectively; n is the number of training samples; Perceptual loss function L p The calculation formula is shown in Equation (3) as follows: Among them, L2 represents the mean square error, and vgg(·) represents the pre-trained VGG16 network; G(·) represents the operation of the generator defogging network, and G(I i ) and J i respectively represent the defogged image generated by the generator and the real fog-free comparison image; the VGG16 network is a network trained on ImageNet; Adversarial loss function L a is also integrated into the loss function to reflect the guiding training role of the discriminator network, and its definition formula is shown in Equation (4): Among them, D(G(I i )) is the probability that the discriminator network determines that the dehazed image G(I i ) generated by the generator is true.

Images