Image dehazing method based on generative adversarial network integrating multi-level features

Abstract

The present invention relates to a generative adversarial network image defogging method that integrates multi-level features. The method is based on an end-to-end defogging algorithm based on deep learning. During downsampling, feature extraction is performed for foggy images of different resolutions respectively, and the extracted different feature maps are learned through the SE-ResNet module to better fit the information between channels and prevent the performance degradation caused by the deepening of the network. The learned multi-level feature maps are spliced ​​to integrate more image features. Then, during upsampling, the down-sampled fused feature maps are put into the SE module for learning to better allocate channel weights and enhance useful features. The learned feature maps are then spliced ​​with the up-sampled feature maps to integrate more image information. Finally, the residual map learned by the network is added to the input foggy image to obtain the final image defogging result. Experimental results show that the image defogging method proposed in the present invention has better defogging performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a generative adversarial network image defogging method integrating multi-level features. Background Art

[0002] In haze weather, the irradiance received by the camera from the scene attenuates along the line of sight. The turbid medium in the atmosphere absorbs the light of the scene image, causing the captured image to lose contrast and color fidelity, reducing the image quality, and directly affecting tasks such as video surveillance, remote sensing imaging, automatic navigation, and target recognition. Therefore, by designing an algorithm model to eliminate the adverse effects of haze factors in the image, the foggy image is restored to a clear image, so that people can obtain more required information from the image. Existing image defogging algorithms are mainly divided into two categories, namely traditional image defogging methods and defogging methods based on deep learning.

[0003] Classic traditional dehazing methods are mainly based on the construction of atmospheric scattering models to obtain dehazed images. However, the overall color of the dehazed images is dark and the image edges are easily distorted, or the sky area in the image is easily ineffective and the computation is large. Since the traditional methods have obvious human interference factors in obtaining parameters, the dehazing effect is not ideal. In recent years, dehazing methods based on deep learning have begun to emerge. Cai et al. [1] proposed a trainable network (DeHazeNet) to estimate image transmittance, that is, input a foggy image and output the corresponding transmittance map through the network. Ren et al. [2] designed a multi-scale convolutional neural network architecture to learn image features (MSCNN). First, the coarse-scale network estimates the rough scene transmittance map, and then the fine-scale network refines the map. Due to the human interference factors of atmospheric light values, networks that no longer estimate transmittance and atmospheric light values ​​separately have gradually emerged, but directly generate dehazing results through convolutional neural networks, that is, input a foggy image and directly output the corresponding dehazing results through the network. Li et al. [3] first proposed an end-to-end image dehazing algorithm (AOD-Net), which transforms the transmittance and atmospheric light value into a parameter and reconstructs the dehazed image based on this parameter. Ren et al. [4] designed an end-to-end network (GFN) based on the encoding-decoding structure. The algorithm first performs three preprocessing operations (i.e., white balance, contrast enhancement, and gamma correction) on the given foggy image. The three preprocessing results are used as network input, and the corresponding feature maps are generated through network learning. The three preprocessing results are fused with the three feature maps to obtain the final image dehazing result.

[0004] End-to-end networks based on deep learning mostly obtain image defogging results by learning graphic features. In recent years, the generative adversarial network (GAN) structure has played a good role in image-to-image conversion. Some researchers have applied this structure to design image defogging algorithms and achieved good results. Therefore, in order to obtain more graphic features to better model foggy graphics, this paper proposes a generative adversarial network image defogging algorithm that integrates multi-level features. Summary of the invention

[0005] The purpose of the present invention is to provide a generative adversarial network image defogging method that integrates multi-level features in order to better remove fog in images and restore image detail information.

[0006] To achieve the above-mentioned purpose, the technical scheme of the present invention is: a generative adversarial network image defogging method integrating multi-level features, embedding the structure of U-Net into the generator of the generative adversarial network, and proposing a generative adversarial network integrating multi-level features, including a generator for generating a defogged image, and a discriminator for discriminating the defogged image and the label image and feeding back the result to the generator; the generator generates the defogged image in the following manner: first, when downsampling, feature extraction is performed for foggy images of different resolutions respectively, and the extracted different feature maps are learned through the SE-ResNet module; then, the learned multi-level feature maps are spliced ​​to fuse more image features; then, when upsampling, the fused feature maps are put into the SE module for learning to better allocate channel weights and enhance useful features; then, the learned feature maps are spliced ​​with the upsampled feature maps to fuse more image information; finally, the residual map learned by the network is added to the input foggy image to obtain the final image defogging result.

[0007] In one embodiment of the present invention, the convolution kernels in the generator are all 3×3.

[0008] In one embodiment of the present invention, in the discriminator, 8 convolutions are used, the convolution kernel of each convolution is 5×5, and the number of channels in each layer is 64, 64, 128, 128, 256, 256, 512 and 512 respectively, which are then folded into a one-dimensional array.

[0009] In one embodiment of the present invention, for three foggy images with resolutions of 256×256, 128×128 and 64×64, respectively, during downsampling, feature extraction is performed on the foggy images with the same resolution of each layer of feature maps, and then the learned feature maps are spliced ​​with the original feature maps of the layer after learning through the SE-ResNet module to integrate richer image features; in the second downsampling layer, the original feature map of the layer in the downsampling is convolved with the foggy image with a resolution of 128×128 and the feature map after learning through the SE-ResNet module is spliced; in the third and fourth downsampling layers, three feature maps are spliced, which are the original feature map of the layer in the downsampling, the feature map after convolution of the foggy image with a resolution of 128×128 and learning through the SE-ResNet module, and the feature map after convolution of the foggy image with a resolution of 64×64 and learning through the SE-ResNet module.

[0010] In one embodiment of the present invention, the SE-ResNet module is a SE module added to the residual block; wherein,

[0011] Residual block structure input x l and output x l+1 The relationship is:

[0012] x l+1 =x l +F(x l ,ω l )

[0013] ω l is the weight of the lth layer, F(·) is the nonlinear transformation;

[0014] The SE module first compresses the input feature map into a one-dimensional vector through global average pooling, and then performs an excitation operation, which is composed in the order of fully connected layer, ReLU layer, fully connected layer, and Sigmoid layer to better build the correlation between channels. Finally, the weight is added to the original features through the Scale operation. The SE module emphasizes valid information and downplays invalid information by weighting the channels, so as to better obtain image features and improve network performance.

[0015] In one embodiment of the present invention, the generator uses mean square error as the loss function, and the loss function formula is:

[0016]

[0017] Among them, n is the number of pixels in the image, y is the defogging result output by the network, is the corresponding label image.

[0018] In one embodiment of the present invention, the discriminator uses binary cross entropy as the loss function, and the discriminator formula is as follows:

[0019]

[0020] Among them, x represents the label image, z represents the foggy image, G(z) represents the defogging result output by the generator, and P data (x) represents the data distribution of the labeled image, P z (z) represents the data distribution of foggy images, E represents the mathematical expectation of these distributions, and D() represents the discriminator.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) The method of the present invention embeds the structure of U-Net into the generator of the generative adversarial network and proposes a generative adversarial network image dehazing method that integrates multi-level features;

[0023] (2) When extracting features, the method of the present invention extracts features from foggy images of different resolutions and fuses multi-level feature maps to obtain more image features;

[0024] (3) The method of the present invention uses SE modules and SE-ResNet modules to further learn image features and better fit the correlation between channels of feature maps to enhance effective information;

[0025] The defogging image generated by the model trained by the method of the present invention has good robustness and can achieve better performance than the other six common algorithms in both foggy and non-foggy conditions. The image defogging method proposed in the present invention can better improve the image defogging performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure is a flow chart of the method of the present invention.

[0027] Figure 2 This is a network structure diagram of the present invention.

[0028] Figure 3 This is the residual block structure of the present invention.

[0029] Figure 4 This is the SE-ResNet block structure of the present invention.

[0030] Figure 5 Image defogging results obtained by seven image defogging algorithms on eight indoor foggy images in SOTS.

[0031] Figure 6 Image defogging results obtained by seven image defogging algorithms on eight outdoor foggy images in SOTS.

[0032] Figure 7 Image dehazing results obtained by seven image dehazing algorithms on six real-world foggy images.

[0033] Figure 8 The image defogging results on the haze-free image: the first row is the haze-free image, and the second row is the image defogging result. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0035] The invention discloses a generative adversarial network image defogging method integrating multi-level features. The structure of U-Net is embedded in a generator of a generative adversarial network, and a generative adversarial network integrating multi-level features is proposed. The method comprises a generator for generating a defogging image and a discriminator for discriminating the defogging image from a label image and feeding back the result to the generator. The generator generates the defogging image in the following manner: first, during downsampling, feature extraction is performed on foggy images of different resolutions respectively, and the extracted different feature maps are learned through a SE-ResNet module; then, the learned multi-level feature maps are spliced ​​to integrate more image features; then, during upsampling, the fused feature maps are put into a SE module for learning to better allocate channel weights and enhance useful features; then, the learned feature maps are spliced ​​with the upsampled feature maps to integrate more image information; finally, the residual map learned by the network is added to the input foggy image to obtain the final image defogging result.

[0036] The following is the specific implementation process of the present invention.

[0037] like Figure 1 The flowchart of the method of the present invention is shown. Inspired by the U-Net and generative adversarial network structures, the present invention proposes a generative adversarial network image defogging method that integrates multi-level features. The network model of the method is as follows: Figure 2 As shown in the figure, the model consists of two parts: a generator and a discriminator. A foggy image is input into the generator, and then the image defogging result is generated. The defogging result and the label image are input into the discriminator for discrimination, and the discrimination result is fed back to the generator for further training.

[0038] 1. Network structure

[0039] The method proposed in the present invention mainly improves the generator. The convolution kernels in the entire generator network are all 3×3. During downsampling, feature extraction is performed on images of three resolutions, namely 256×256, 128×128 and 64×64. The step size of the three convolutions for the image with a resolution of 128×128 is 1, and the number of channels after convolution is 64, 128 and 256 respectively. The step size of the two convolutions for the image with a resolution of 64×64 is 1, and the number of channels after convolution is 128 and 256 respectively. After convolution, the images with a resolution of 128×128 and 64×64 are both learned by two SE-ResNet modules and then spliced ​​with the remaining feature maps of the layer. Downsampling process ( Figure 2 The number of channels of each layer of feature maps is 64, 128, 256 and 512 respectively. The feature map of the last layer is learned by 10 SE-ResNet modules. Then, upsampling is performed three times, and the parameter scale_factor is set to 2. The first two upsampling processes ( Figure 2 The feature maps (indicated by dotted lines in the figure) are all concatenated with the concatenated feature maps learned by the SE module corresponding to the downsampling. Finally, the residual map learned by upsampling is added to the input image to obtain the image defogging result.

[0040] In the discriminator of the method of the present invention, 8 convolutions are used, the convolution kernel of each convolution is 5×5, and the number of channels in each layer is 64, 64, 128, 128, 256, 256, 512 and 512. Then it is folded into a one-dimensional array.

[0041] 2. Downsampling

[0042] In order to obtain richer image features, feature extraction is performed on foggy images of three different resolutions to fuse more feature maps, namely 256×256, 128×128 and 64×64. Feature extraction is performed on foggy images with the same resolution of feature maps in each layer of downsampling, and then the learned feature maps are spliced ​​with the original feature maps of the layer after learning through the SE-ResNet module to fuse richer image features. In the second layer of downsampling, we splice the original feature map of the layer in the downsampling and the feature map after convolution of the foggy image with a resolution of 128×128 and learning through the SE-ResNet module. In the third and fourth layers of downsampling, we splice three feature maps, namely the original feature map of the layer in the downsampling, the feature map after convolution of the foggy image with a resolution of 128×128 and learning through the SE-ResNet module, and the feature map after convolution of the foggy image with a resolution of 64×64 and learning through the SE-ResNet module.

[0043] 3. SE-ResNet module

[0044] As the number of network layers increases, network performance will degrade, and the residual network can prevent this degradation. Figure 3 This is an example of a residual block structure, where the jump connection structure connects the input and output, combining features of different resolutions, that is, adding shallow high-resolution features to deep low-resolution features.

[0045] Residual block structure input x l and output x l+1 The relationship is:

[0046] x l+1 =x l +F(x l ,ω l )

[0047] ω l is the weight of the lth layer, F(·) is the nonlinear transformation;

[0048] The SE-ResNet module adds the SE (Squeeze-and-Excitation) module to the residual block. Figure 4 This is the structural diagram of the SE-ResNet module. Figure 4 H, W and C in the image represent the height, width and number of channels of the image, respectively, and r represents the reduction coefficient, which is used to reduce the number of channels. In the experiment of the present invention, r is set to 16. The SE module first compresses the input feature map into a one-dimensional vector through global average pooling, and then performs an excitation operation, which is composed in the order of a fully connected layer, a ReLU layer, a fully connected layer, and a Sigmoid layer to better construct the correlation between channels. Finally, the weight is added to the original feature through the Scale operation (dot multiplication operation). The SE module emphasizes valid information and downplays invalid information by weighting the channels, so as to better obtain image features and improve network performance.

[0049] 4. Loss Function

[0050] The loss function determines the direction of network training descent, and a suitable loss function can improve network performance. The present invention uses mean square error as the loss function of the network generator. The loss function formula is:

[0051]

[0052] Among them, n is the number of pixels in the image, y is the defogging result output by the network, is the corresponding label image.

[0053] The present invention uses binary cross entropy as the loss function of the discriminator, and the discriminator formula is as follows:

[0054]

[0055] Among them, x represents the label image, z represents the foggy image, G(z) represents the defogging result output by the generator, and P data (x) represents the data distribution of the labeled image, P z (z) represents the data distribution of foggy images, E represents the mathematical expectation of these distributions, and D() represents the discriminator.

[0056] 5. Experimental data and evaluation

[0057] In order to verify the effectiveness of the method, qualitative and quantitative experimental results will be used to confirm it. The method proposed in this invention will be compared with six existing image dehazing algorithms, namely: DCP algorithm, CAP algorithm, DeHazeNet algorithm, MSCNN algorithm, AOD-Net algorithm, and GFN algorithm. The first two are traditional image dehazing algorithms, and the last four are algorithms based on deep learning like the method proposed in this invention. The RESIDE dataset and evaluation indicators used for training and testing the network are used to qualitatively and quantitatively analyze the network test results, and finally the robustness of the method is tested.

[0058] The quantitative comparison of the present invention uses two common evaluation indicators of image defogging algorithms: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The peak signal-to-noise ratio is an objective standard for measuring image distortion. The higher the value, the better the image quality. The mathematical formula of the peak signal-to-noise ratio is:

[0059]

[0060] Among them, W and H are the width and height of the image respectively, G represents the label image, and y represents the image dehazing result output by the network.

[0061] SSIM is an indicator to measure the similarity between two images. The higher the SSIM value, the better the image quality. The mathematical formula of SSIM is:

[0062]

[0063] Among them, μ and δ represent the mean and variance of the image respectively, δ G,y represents the covariance between the haze-free image G and the image dehazing result y generated by the network, and c1 and c2 are constants.

[0064] 6. Quantitative comparison

[0065] Tables 1 and 2 give the quantitative evaluation results of seven image dehazing algorithms on indoor and outdoor images, respectively. The highest values ​​are indicated in bold.

[0066] Table 1 Quantitative comparison of average PSNR and SSIM values ​​of indoor foggy images of SOTS dataset

[0067]

[0068] Table 2 Quantitative comparison of average PSNR and SSIM values ​​of outdoor foggy images of SOTS dataset

[0069]

[0070]

[0071] 7. Qualitative comparison

[0072] In order to further qualitatively compare the dehazing effects of different algorithms, Figure 5 The image defogging results obtained by seven image defogging algorithms on four indoor light fog images (the first four) and four indoor dense fog images (the last four) in SOTS are shown. The seven image defogging algorithms can remove image fog well on the light fog images, but the DCP algorithm will have obvious distortion, and the images after defogging by the CAP and DeHazeNet algorithms are obviously darker than the labeled images. The defogging images of the MSCNN and AOD-Net algorithms have obvious fog, while the CAP and DeHazeNet algorithms have less residual fog compared with the first three algorithms, but it is still obvious. The defogging images of GFN and the method proposed in the present invention are cleaner than the first five. The method proposed in the present invention has better defogging effect than the six comparison algorithms.

[0073] Figure 6 The image defogging results obtained by seven image defogging algorithms on eight outdoor foggy images in SOTS are shown. The defogging method proposed in the present invention is closer to the label image and achieves better defogging performance, while the DCP algorithm has obvious distortion. The CAP and DeHazeNet algorithms can remove fog from the image well, but the overall apparent color of the defogging results is darker than that of the label image. The defogging images of the MSCNN and AOD-Net algorithms have obvious fog and the defogging effect is average. The defogging images using the GFN and the present invention methods have less residual fog.

[0074] Figure 7 The image defogging results obtained by seven image defogging algorithms on six real-world foggy images are shown. The results show that the method of the present invention can achieve a good balance between removing image fog and color saturation of defogging images. The DCP algorithm and the CAP algorithm also have obvious distortion, and the defogging images of the MSCNN and AOD-Net algorithms have obvious fog. The degree of fog removal of DeHazeNet, GFN and the method of the present invention is good, but the image after defogging by the GFN algorithm is obviously too dark.

[0075] 8. Robustness Testing

[0076] In order to verify that the method of the present invention has no adverse effect on the fog-free image, the present invention selects two indoor images and two outdoor images for comparison. Figure 8 The defogging results of the method of the present invention on four fog-free images are shown, the first two are indoor images, and the latter two are outdoor images. Figure 8 It can be seen that the defogging results of the defogging method proposed in the present invention for both indoor and outdoor images are very close to the original images, that is, the method of the present invention has no obvious adverse effect on the fog-free image.

[0077] References:

[0078] [1]He KM, Sun J, Tang X O.Single image haze removal using dark channel prior[C] / / Proceedings of the IEEE Conference on ComputerVision and PatternRecognition,2009:1956-1963.

[0079] [2] Zhu Q, Mai J, Shao L.AFast Single Image Haze Removal Algorithm UsingColorAttenuation Prior[J]. IEEE Transactions on Image Processing, 2015, 24(11): 3522-3533.

[0080] [3]Cai B, Xu

[0081] [4]Ren WQ, Liu S, Zhang H, et al.Single Image Dehazing via Multi-scaleConvolutional Neural Networks[C] / / Proceedings of the European Conference on Computer Vision (ECCV), 2016:154-169.

[0082] [5]Li BY, Peng XL, Wang ZY, et al. AOD-Net: All-In-One Dehazing Network [C]. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017: 4770-4778.

[0083] [6]Ren WQ, Ma L, Zhang JW, et al. Gated Fusion Network for Single ImageDehazing[C]. Proceedings of the IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2018: 3253-3261.

[0084] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A generative adversarial network image dehazing method integrating multi-level features, characterized in that: The structure of U-Net is embedded in the generator of the generative adversarial network, and a generative adversarial network integrating multi-level features is proposed, including a generator for generating defogging images and a discriminator for distinguishing defogging images from label images and feeding the results back to the generator; the generator generates defogging images in the following way: first, when downsampling, feature extraction is performed on foggy images of different resolutions respectively, and the extracted different feature maps are learned through the SE-ResNet module; then, the learned multi-level feature maps are spliced ​​to integrate more image features; Then, during upsampling, the fused feature map is put into the SE module for learning to better allocate channel weights and enhance useful features; then, the learned feature map is spliced ​​with the upsampled feature map to fuse more image information; finally, the residual map learned by the network is added to the input foggy image to obtain the final image defogging result; for three foggy images with resolutions of 256×256, 128×128 and 64×64, respectively, during downsampling, feature extraction is performed on foggy images with the same resolution of each layer of feature map, and then the learned feature map is combined with the input foggy image after learning through the SE-ResNet module. The original feature maps of the downsampling layer are spliced ​​to integrate richer image features; in the second downsampling layer, the original feature map of the downsampling layer is convolved with the feature map after the foggy image with a resolution of 128×128 and the SE-ResNet module is learned; in the third and fourth downsampling layers, three feature maps are spliced, namely the original feature map of the corresponding layer in the downsampling, the feature map after the foggy image with a resolution of 128×128 and the SE-ResNet module, and the feature map after the foggy image with a resolution of 64×64 and the SE-ResNet module is learned.

2. The method for image dehazing by generative adversarial network integrating multi-level features according to claim 1, characterized in that: The convolution kernels in the generator are all 3×3.

3. The method for image dehazing by generative adversarial network integrating multi-level features according to claim 1, characterized in that: In the discriminator, 8 convolutions are used, the convolution kernel of each convolution is 5×5, and the number of channels in each layer is 64, 64, 128, 128, 256, 256, 512 and 512 respectively, which are then folded into a one-dimensional array.

4. The method for image dehazing by generative adversarial network integrating multi-level features according to claim 1, characterized in that: The SE-ResNet module is a module that adds an SE module to the residual block; wherein, Residual block structure input x l and output x l+1 The relationship is: x l+1 =x l +F(x l ,ω l ) ω l is the weight of the lth layer, F(·) is the nonlinear transformation; The SE module first compresses the input feature map into a one-dimensional vector through global average pooling, and then performs an excitation operation, which is composed in the order of fully connected layer, ReLU layer, fully connected layer, and Sigmoid layer to better build the correlation between channels. Finally, the weight is added to the original features through the Scale operation. The SE module emphasizes valid information and downplays invalid information by weighting the channels, so as to better obtain image features and improve network performance.

5. The method for image dehazing by generative adversarial network integrating multi-level features according to claim 1, characterized in that: The generator uses mean square error as the loss function, and the loss function formula is: Among them, n is the number of pixels in the image, y is the defogging result output by the network, is the corresponding label image.

6. The method for image defogging by generative adversarial network integrating multi-level features according to claim 1, characterized in that: The discriminator uses binary cross entropy as the loss function, and the discriminator formula is as follows: Among them, x represents the label image, z represents the foggy image, G(z) represents the defogging result output by the generator, and P data (x) represents the data distribution of the labeled image, P z (z) represents the data distribution of foggy images, E represents the mathematical expectation of these distributions, and D() represents the discriminator.

Images