Image defogging method based on improved generative adversarial network

Abstract

The invention discloses an image defogging method based on an improved generative adversarial network, and the method comprises the steps: constructing the improved generative adversarial network: selecting a CycleGAN network as a basic structure, introducing a multi-depth feature extraction module to obtain feature maps of different semantic information, and employing a haze sensing unit HPB to learn haze features, a CBAM mixed attention mechanism is fused to realize feature enhancement of the key area; for multi-feature selection, a self-adaptive weight module is adopted to carry out emphasized screening from a part dimension and an overall dimension; carrying out defogging processing on the haze image by using the constructed improved generative adversarial network; according to the method, the problems that the defogging performance is poor in an area with large scene depth, and the fidelity of colors and the reduction degree of details need to be improved in an existing method can be solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and image processing, and in particular relates to an image defogging method based on an improved generative adversarial network. Background Art

[0002] With the rapid development of my country's economy and informatization, computer vision technology has been widely used in various fields. Video images, as the basic input form of computer vision, play a key role in the realization of related tasks. However, video images are easily affected by the external environment, especially the rapid advancement of my country's industrialization and urbanization. The large amount of industrial waste gas and traffic exhaust emissions have led to frequent smog. The larger particles suspended in smog weather seriously affect the absorption and scattering of light, resulting in image distortion, blurring, reduced contrast, obvious quality degradation, and missing details. These problems affect the acquisition of image information, reduce image quality and availability, and affect the application of video surveillance in target detection, vehicle model recognition, intelligent driving and other scenarios.

[0003] In order to solve the above problems, image defogging technology came into being. The purpose of image defogging is to reduce or eliminate the haze in the image, increase the image clarity, obtain more effective image information, restore the real scene color of the image, and obtain a clear image. Defogging technology can effectively improve the image quality, provide clear input data for abnormal behavior detection, vehicle recognition, pedestrian detection, geographic survey, disaster monitoring, remote sensing mapping and other related tasks, and improve the accuracy and reliability of subsequent classification, detection and recognition of the machine. Therefore, how to reduce or eliminate the haze in the image through image processing technology and obtain a clear image has become a current research hotspot and has important practical significance.

[0004] According to different dehazing principles, image dehazing algorithms can be divided into three categories: image enhancement methods, image restoration dehazing and deep learning model dehazing.

[0005] The deep learning-based algorithm can remove haze thoroughly and the visual effect of the dehazed image is good, but it has the shortcomings of dependence on data sets, low scene universality and poor robustness; most of the existing studies rely on pairs of foggy and fog-free images, but in actual work, it is difficult to obtain a large number of matching image pairs due to various factors; the haze distribution of foggy images is uneven, and different features have different effects on image dehazing. The dehazing algorithm treats feature channels equally and cannot fully explore the correlation of feature channels, which limits the feature expression ability of the network. Summary of the invention

[0006] In view of this, the present invention provides an image dehazing method based on an improved generative adversarial network, which can solve the problem that the existing methods have poor dehazing performance in areas with large scene depth, and the color fidelity and detail restoration degree need to be improved.

[0007] The technical solution for implementing the present invention is as follows:

[0008] An image dehazing method based on an improved generative adversarial network is proposed. The improved generative adversarial network is constructed. The CycleGAN network is selected as the basic structure, and a multi-depth feature extraction module is introduced to obtain feature maps of different semantic information. The haze perception unit HPB is used to learn haze features, and the CBAM hybrid attention mechanism is integrated to enhance the features of key areas. For the selection of multiple features, an adaptive weight module (AWM) is used to perform focused screening from two dimensions: part and whole. The haze image is dehazed using the constructed improved generative adversarial network.

[0009] Furthermore, the network structure of U-Net++ is adopted as the overall architecture of the feature extraction part.

[0010] Furthermore, CBAM contains two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in sequence, the recalibrated features will be obtained. That is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features. The spatial channel coefficient is obtained by passing through the spatial attention module, and multiplied with the intermediate features to obtain the output features.

[0011] Furthermore, the structure of the haze perception unit HPB, firstly, adopts a method of depth separable convolution and a mixture of different convolutions to reduce the number of network parameters and expand the depth of the network. Secondly, in order to focus more on the foggy area, a global perception module is proposed, which performs global pooling in the horizontal and vertical directions, and obtains the global perception feature map by splitting, learning and merging. Then, the convolution is replaced by a fully connected method to achieve the change of the number of channels. Compared with the convolution that only focuses on local information, the fully connected method can obtain the overall information of the image, and adopts an anti-bottleneck structure to expand the vector 4 times in the middle layer, introducing a higher dimension than the input and output, thereby increasing the depth and complexity of the network and enhancing the expression ability of the c model. Considering that the network is prone to overfitting and network degradation during training, the strategies of layer scaling and random subpath deletion are proposed to solve them. Finally, the jump connection method is adopted to increase the interaction between input features and output features, so that the network can learn the characteristics of haze more easily and improve the fidelity of the image.

[0012] Furthermore, the constructed improved generative adversarial network integrates edge loss and image brightness loss.

[0013] Beneficial effects:

[0014] 1. The present invention introduces a multi-depth feature extraction module to obtain feature maps of different semantic information, and uses the haze perception unit HPB to learn haze features, integrating the CBAM hybrid attention mechanism. For the selection of multiple features, an adaptive weight module is used to enable the network to have better haze removal capabilities.

[0015] 2. The present invention effectively avoids the problem that the generated defogging image is not completely defogging and has poor color fidelity and detail restoration.

[0016] 3. The present invention integrates edge loss and image brightness loss, so that the network model has better color fidelity and detail restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the CycleGAN network principle described in the present invention.

[0018] Figure 2 This is a schematic diagram of the improved generator described in the present invention.

[0019] Figure 3 Schematic diagram of the multi-depth feature extraction module.

[0020] Figure 4 This is a schematic diagram of the haze sensing unit structure. DETAILED DESCRIPTION

[0021] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0022] The present invention provides an image defogging method based on an improved generative adversarial network, comprising the following steps:

[0023] 1. CycleGAN network principle

[0024] As a derivative network of GAN, CycleGAN inherits powerful generation capabilities. Its learning and training does not rely on any similarity relationship between input and output. This advantage enables it to be trained on unpaired data sets and is more practical. Therefore, the present invention uses CycleGAN as the basic structure. CycleGAN consists of two generator networks and two discriminator networks, and divides the entire network into two GAN network systems through cycle consistency loss. The logical framework of CycleGAN is shown in the attached figure. Figure 1 As shown:

[0025] GAN is a generative adversarial network, where adversarial refers to the game between the generator and the discriminator. The goal of the generator is to make the generated samples as close to the real data as possible, and the goal of the discriminator is to accurately judge the generated samples as false and the real samples as true. The input z is usually random noise that obeys a certain distribution. The generator G generates a false sample G(z). The discriminator D judges whether G(z) obeys the distribution of the real data. If G(z) is "true", the output is 1, indicating that the generator can fit the real sample, otherwise the output is 0. The generator G is equivalent to a function, which is usually modeled as a neural network. In theory, neural networks can fit any function. The purpose is to realize the mapping of random input vectors to target distributions. The discriminator is actually equivalent to a classifier, and the discriminator is usually modeled as a neural network. The CycleGAN model consists of two generators G and F and two discriminators D x , D y The generator F converts the output image of the generator G into the source image domain. Similarly, the generator G converts the output image of the generator F into the target image domain. The discriminator D x Determine whether the generated image of F belongs to the source domain X, the discriminator D y Determine whether the generated image of G belongs to the target domain Y. CycleGAN measures the loss of the image in such a ring transformation through cycle-consistency loss.

[0026] 2. Improve the CycleGAN generator network structure

[0027] Improve the cyclegan generator network structure G as follows Figure 2 As shown:

[0028] (1) Multi-depth feature extraction module

[0029] Feature extraction is a key part of the entire generator network. In previous studies, some works have used U-Net++ as a feature extraction network and achieved good results. Its specific structure is as follows: Figure 3 As shown. As can be seen from the figure, the structure consists of four U-shaped networks with increasing depths, and uses dense connections to extract features of different depths. Generally speaking, networks of different depths have different expressive capabilities. The shallow network feature receptive field is small and the local information is rich, while the deep network feature receptive field is large and the global information is rich. In the image defogging task, it is necessary to use sufficient global information to restore the main body of the image, and combine the delicate local information to restore the image details, so as to achieve the ideal defogging effect. Therefore, the present invention adopts the network structure of U-Net++ as the overall architecture of the feature extraction part.

[0030] Although the multi-depth feature extraction module can effectively extract high-level semantic information and shallow detail information, some information is lost in the haze image due to the presence of masking, making it difficult to restore. In order to learn this part of the features, improve the ability to remove haze, and enhance image restoration, the present invention proposes a haze perception unit HPB, the specific structure of which is as follows: Figure 4 As shown in the figure. First, the depth-separable convolution and the mixture of different convolutions are used to reduce the number of network parameters and expand the depth of the network. Secondly, in order to focus more on the foggy area, a global perception module is proposed to perform global pooling in the horizontal and vertical directions, and obtain the global perception feature map by splitting, learning and merging. Then, the convolution is replaced by a fully connected method to achieve the change of the number of channels. Compared with the convolution that only focuses on local information, the fully connected method can obtain the overall information of the image, and adopts an anti-bottleneck structure to expand the vector 4 times in the middle layer, introducing a higher dimension than the input and output, thereby increasing the depth and complexity of the network and enhancing the expression ability of the c model. Considering that the network is prone to overfitting and network degradation during training, the strategies of layer scaling and random subpath deletion are proposed to solve the problem. Finally, the skip connection method is used to increase the interaction between input features and output features, so that the network can learn the characteristics of haze more easily and improve the fidelity of the image. In order to ensure the stability of the gradient transmission process, the GELU activation function is used instead of the commonly used Relu activation function in the selection of nonlinear activation functions. Compared with the Relu activation function, the GELU activation function has a smoother gradient at zero. In the selection of the normalization function, the LN layer is used instead of the BN layer. Since the BN layer is normalized from the batch dimension, the effect is not ideal when the batch is very small, while the LN layer is normalized from the channel dimension and is not affected by the batch size.

[0031] (2) CBAM Hybrid Attention Mechanism

[0032] The CBAM hybrid attention mechanism is a module that can focus on important local features of feature images. The core idea is to simulate the behavior of human selective attention in vision by assigning weights. It assigns weights to feature images in space and channels through adaptive learning, so that the output feature map is more inclined to the target area of ​​focus, thereby enhancing the features of the key areas. CBAM consists of two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in turn, the recalibrated features will be obtained. That is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features. The spatial attention module obtains the spatial channel coefficient, which is multiplied with the intermediate features to obtain the output features. The length, width and number of channels of the output features remain unchanged compared to the input features. G consists of 3 downsampling modules, 3 CBAM modules, 9 residual modules and 3 upsampling modules. Among them, the downsampling module, CBAM module and residual module belong to the encoder part, which are responsible for extracting the features of the input image. The upsampling block belongs to the decoder part, which is responsible for deconvolution of the acquired features to generate the output image.

[0033] (3) Adaptive Weight Module

[0034] The five branch features from the multi-depth feature extraction module are used to restore the fog-free image. Considering that the features of different paths carry different information and have different effects on the defogging effect, if each path is treated equally and the features are connected in series by hard fusion, the network expression will be limited, resulting in the problem of low utilization of effective information. In this regard, the present invention proposes an adaptive weight module AWM, which uses CBAM and an adaptive weight mechanism to screen the required features.

[0035] Where F represents the feature map after feature extraction, f represents the feature map after screening, and M c Represents the selection from the image channel dimension. First, the feature map is filtered from the channel dimension by average pooling and maximum pooling. Then, the multi-layer perceptron MLP is used to further integrate the channel information. Finally, the weighted filtering map M of the channel dimension is obtained through Sigmoid activation. c (F).

[0036] According to the process of multiple weight values ​​and peak signal-to-noise ratio changing with training rounds, during the network training process, w 1 、w 2 The weight is on the rise, w 3 、w 4 、w 5 The weights are decreasing, where w 2 The proportion is relatively large, w 5 This shows that in w2 In the corresponding Bath_2 path, the network of this depth has better haze removal ability than networks of other depths.

[0037] 3. Determine the network structure

[0038] The definition of the discriminant network D is a binary classification network that classifies the input image into true or false, and the output value is 0 and 1. In actual training, in order to better adapt the formula of the adversarial loss function, converge the model faster, and improve the capabilities of the generation network and the discriminant network, the output of the discriminant network is often a single probability value for predicting the image to be true.

[0039] 4. Loss Function

[0040] Adversarial loss and cycle consistency loss only ensure that the generated image is as consistent as possible with the original image in terms of data distribution and style, while ignoring the feature consistency in the image. Generally, high-quality images can be generated by optimizing the loss function. When dehazing based on the CycleGAN network, the edge loss and image brightness loss are integrated to achieve better color fidelity and detail restoration.

[0041] There are two loss functions in the model training process, adversarial loss and cycle consistency loss. The former makes the data distribution generated by the generator as close to the real data distribution as possible, and the latter is used to ensure that the two generators can achieve mutual inversion, that is, iterate back to themselves. Let X and Y be the visible light data set and infrared data set, x represents the image sample in the visible light data set, y represents the image sample in the infrared data set, for the generator G:X→Y and its discriminator D Y , the adversarial loss function is:

[0042]

[0043] Where: E is the mathematical expectation; P data is the distribution of foggy image data; D Y (y) represents the sample y in the real fog-free database Y in the discriminator D Y The closer the score is to 1, the more realistic the discriminator thinks the defogging is. G(x) is the infrared simulated image generated by the generator based on the foggy x and the image with no fog in Y. Y (G(x)) represents the discriminator D Y According to the score obtained by the generated haze-free simulated image, if D Y The lower the authenticity of the generated fog-free simulation image, the higher the Y The closer (G(x)) is to 0, the smaller the YThe closer (G(x)) is to 1. When the discriminator DY is stronger, that is, it can better distinguish the real fog-free image from the fog-free simulated image generated by the generator, the larger the loss function value is. The generator G hopes to generate as realistic fog-free simulated images as possible to deceive the discriminator, that is, it hopes that the loss function value is as small as possible. Similarly, the generator F:Y→X and its discriminator D x The same is true for the adversarial loss function:

[0044]

[0045] The cycle consistency loss function is the core of CycleGAN and the core of realizing the conversion between unpaired images. In order to ensure cycle consistency as much as possible, Lcyc is set to two similar structures:

[0046]

[0047] Where: ||·||1 represents the l-norm. In summary, the total loss function of the CycleGAN network is:

[0048] L CycleGAN =L GAN (G,D Y ,X,Y)+L GAN (F,D x ,Y,X)+λL cyc (G,F)

[0049] Where: λ is the weight for balancing adversarial loss and cycle-consistent loss, usually 10

[0050] The shape of the target object in the image has boundaries, and the edge outlines the contour of the target. The directional gradient histogram feature is a feature descriptor commonly used in the field of computer vision and image processing to detect objects. The edge information of the target is obtained by counting the local gradient direction of the image. First, the image must be preprocessed, grayscaled and gamma corrected, the image contrast must be adjusted, the impact of local shadows and illumination changes in the image must be reduced, and noise interference must be suppressed; the pixel gradient is calculated to obtain the contour information of the target, the gradient histogram is calculated, and the HOG feature vector of the image is calculated after normalization. The distance between the synthetic infrared image and the real infrared image in terms of structural shape is measured based on the HOG feature.

[0051]

[0052] Where: GRA represents the edge feature vector of the image

[0053] Image intensity refers to the size of the image pixel value of a single channel, such as the pixel value of each point in a grayscale image. In the RGB color space, it can be understood that the three channels R, G, and B each have an image intensity. That is to say, an RGB image contains three image intensities. Infrared images are based on the principle of reflection imaging, and rely on thermal infrared sensors to receive radiation reflected from objects for imaging. Therefore, the intensity value of the infrared image must be related to the amount of radiation reflected by the object, that is, the temperature of the object at that time.

[0054] S=R*0.299+G*0.587+B*0.114

[0055] By adding image intensity to the objective function, the color restoration of the generated haze-free image can be improved to a certain extent.

[0056]

[0057] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image dehazing method based on an improved generative adversarial network, characterized in that: Construct an improved generative adversarial network: select the CycleGAN network as the basic structure, introduce a multi-depth feature extraction module to obtain feature maps of different semantic information, use the haze perception unit HPB to learn haze features, and integrate the CBAM hybrid attention mechanism to enhance the features of key areas; for the selection of multiple features, use the adaptive weight module to perform focused screening from both the partial and overall dimensions; use the constructed improved generative adversarial network to dehaze haze images.

2. The image defogging method according to claim 1, characterized in that: The network structure of U-Net++ is used as the overall architecture of the feature extraction part.

3. The image defogging method according to claim 1 or 2, characterized in that: CBAM consists of two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in sequence, they will finally obtain recalibrated features, that is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features. The spatial channel coefficient is obtained by passing through the spatial attention module, and multiplied with the intermediate features to obtain the output features.

4. The image defogging method according to claim 3, characterized in that: The structure of the haze perception unit HPB is as follows: first, a depth-separable convolution and a mixture of different convolutions are adopted to reduce the network parameters and expand the network depth; second, in order to focus more on the foggy area, a global perception module is proposed, global pooling is performed in the horizontal and vertical directions, and a global perception feature map is obtained by splitting, learning and merging; then, a full connection method is used to replace the convolution to achieve a change in the number of channels. Compared with the convolution that only focuses on local information, the full connection method can obtain the overall information of the image, and an anti-bottleneck structure is adopted to expand the vector 4 times in the middle layer, introducing a higher dimension than the input and output, thereby increasing the depth and complexity of the network and enhancing the expression ability of the c model; considering that the network is prone to overfitting and network degradation during training, a layer scaling and random subpath deletion strategy is proposed to solve the problem; finally, a skip connection method is adopted to increase the interaction between input features and output features, so that the network can learn the characteristics of haze more easily and improve the fidelity of the image.

5. The image defogging method according to claim 1, characterized in that: The constructed improved generative adversarial network integrates edge loss and image brightness loss.

Images