A method and device for image dehazing based on SqueezeNet generative adversarial network

Abstract

The present invention discloses a method and device for image defogging based on a SqueezeNet-based generative adversarial network. The method comprises: inputting a collected foggy image into a constructed SqueezeNet-based generative adversarial network model, and outputting a defogging fog-free image; wherein the SqueezeNet-based generative adversarial network model comprises a generator network and a discriminator network. The present invention extracts fog density features through a feature extraction network SqueezeNet, so that the defogging network can automatically adjust the weights of areas with different concentrations, more effectively separate haze and image content, and achieve non-uniform processing of haze of different concentrations in real scenes. The improved attention block enables the defogging network to extract key features of the image, thereby improving the performance of the defogging network. The introduction of perceptual loss helps to recover more image detail information, avoiding the problem of loss of image spatial details caused by traditional convolutional neural networks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and device for image defogging based on a generative adversarial network (GAN) using SqueezeNet. Background Art

[0002] In recent years, with the development of artificial intelligence, computer vision has been widely applied to various areas of our lives, such as image recognition and image segmentation. These high-level image processing tasks place high demands on the quality of the input images. However, in autumn and winter, due to large temperature fluctuations, mountain highways are prone to fog. This fog creates hazy conditions and low visibility, ranging from tens to tens of meters. When fog appears on highways, traffic accidents are very likely to occur if a vehicle enters, especially in heavy traffic. The scene light information reaching the imaging device is impaired, resulting in blurred contrast and color distortion in the captured images. This affects the collection of highway traffic conditions, the prediction of traffic anomalies, and the determination of accident responsibility. Therefore, research on defogging algorithms for surveillance images is of great significance.

[0003] Currently, image dehazing algorithms can be mainly divided into three types: the first type is based on image enhancement. This method does not consider the cause of image degradation and directly improves the contrast of foggy images. This is prone to problems such as over-enhancement or loss of image information. The quality of the restored fog-free image is average, and the robustness of processing complex image content is very low. The second type is based on image restoration. This method is based on the atmospheric scattering model and uses various methods to estimate the parameters in the model. It then inverts and solves the original image before degradation. This method makes the processed image clearer and more natural, with less loss of detail. However, the dehazing effect is related to the selection of model parameters. Inaccurate parameters will directly affect the effect of the restored image. At the same time, this method requires manual summary of prior knowledge of the image and design of image features, and lacks universality for complex scenes.

[0004] In addition, if the parameter estimation is inaccurate, it will directly affect the final dehazing effect. The third type is the method based on deep learning. In recent years, due to the success of deep learning in the field of high-level image processing, some scholars have applied it to the field of image dehazing and achieved good results. This type of method does not require manual design of feature extractors, but learns the characteristics of haze through the feature extraction ability of neural networks, thereby achieving better image dehazing effects. However, the images restored by existing dehazing neural networks still have the problem of detail loss. For example, AOD-Net, FAMED-Net and DCPDN, ​​these dehazing networks failed to effectively pay attention to haze distribution information and design corresponding strategies when designing the network structure, resulting in artifacts and haze residuals in the dehazed images in some special scenes. Summary of the Invention

[0005] In order to address the deficiencies in the prior art, the present invention provides a method and device for image defogging based on a generative adversarial network (GAN) using SqueezeNet, which has the advantages of good image defogging effect, less loss of image details after defogging, and the model training process is independent of the noise distribution of the data set.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, an image defogging method is provided, comprising: collecting a foggy image; inputting the collected foggy image into a constructed generative adversarial network model based on SqueezeNet, and outputting a defogged fog-free image; wherein the generative adversarial network model based on SqueezeNet includes a generator network and a discriminator network.

[0008] Furthermore, the generator network includes a feature extraction network SqueezeNet and an image reconstruction network connected in sequence; the discriminator network includes multiple convolutional layers, fully connected layers, and sigmoid activation layers connected in sequence; the feature extraction network SqueezeNet learns fog density features from foggy images, which is used for the image reconstruction network to automatically adjust the weights of different concentration areas, thereby achieving non-uniform processing of different concentrations of haze in real scenes; the feature extraction network SqueezeNet includes a data input layer, a convolutional layer, a maximum pooling layer, three Fire modules, a maximum pooling layer, four Fire modules, a maximum pooling layer, a Fir e module, convolution layer; the data input layer is used to receive the input foggy image, and the input foggy image is compressed and expanded by the Fire module. The resolution of the output feature map remains unchanged, and the number of channels changes. For example, the size of the input feature map is H*W*M, and the size of the output feature map becomes H*M*(e1+e3); each of the Fire modules includes a squeeze layer and an expand layer; the squeeze layer is a convolution layer containing a 1×1 convolution kernel, and the expand layer is a convolution layer containing 1×1 and 3×3 convolution kernels; in the expand layer, the feature maps obtained by the 1×1 and 3×3 convolutions are connected, and each convolution layer adopts the ReLu activation function.

[0009] Furthermore, the feature extraction network SqueezeNet is integrated with an attention block, which includes an upper branch and a lower branch, wherein the upper branch adopts two convolutional layers and two LeakyReLU layers to obtain pixel features of input information; the lower branch adopts an average pooling layer, a maximum pooling layer, a convolutional layer and a LeakyReLU layer to obtain channel features of input information; the data obtained by the upper branch and the lower branch are added pixel by pixel, and the Sigmoid function is used to configure corresponding weight parameters for the channel and pixel, and then the formula (1) is used to multiply the image obtained after processing by the feature extraction network SqueezeNet pixel by pixel to realize pixel-by-pixel weighted processing of the foggy image, thereby obtaining a better processing effect on the non-uniformly distributed foggy image.

[0010] M(F)=σ((M P (F)+M C (F)) (1)

[0011] Among them, F is the input feature map obtained from the training sample set, M(F) is the final weight parameter, and M P (F) is the pixel weight parameter, M C (F) is the channel weight parameter, and σ is the Sigmoid activation function.

[0012] Furthermore, the image reconstruction network adjusts the feature maps output by the feature extraction network to a consistent size through deconvolution, then connects them into one feature map, and reconstructs the image through deconvolution, activation and element-wise addition fusion.

[0013] Furthermore, the loss function in the SqueezeNet-based generative adversarial network model includes:

[0014] Added perceptual loss function for:

[0015]

[0016] Among them, j represents the jth layer of the network, CjHjWj represents the size of the feature map of the jth layer in the network, represents the output feature of the jth layer of the real fog-free image, Represents the output features of the jth layer of the fog-free image estimated by the network;

[0017] Discriminator loss function L1:

[0018]

[0019] Among them, D(x i ) represents the discriminant result of the discriminator for the i-th image in the training sample set, D(G(zi )) represents the discrimination result of the discriminator for the i-th generated image generated by the generator, and m is the number of training samples;

[0020] Generator loss function L2:

[0021]

[0022] Among them, D(G(z i )) represents the discrimination result of the discriminator for the i-th generated image generated by the generator, and m is the number of training samples;

[0023] Furthermore, the training method of the SqueezeNet-based generative adversarial network model includes: initializing the weight parameters of the network at each stage; for each sample i, first using forward propagation to calculate the total error, then using backpropagation to calculate the partial derivatives of each weight parameter, and finally updating the weight parameter according to formula (5); repeating this step until the result of each loss function reaches the set value,

[0024]

[0025] Among them, α is the learning rate, W is the weight parameter after update, and W0 is the weight parameter before update. Calculated by the back-propagation algorithm.

[0026] In a second aspect, an image defogging device is provided, comprising: an image acquisition module for acquiring foggy images; a defogging module for inputting the acquired foggy images into a generative adversarial network model constructed based on SqueezeNet, and outputting a defogged fog-free image; wherein, the generative adversarial network model based on SqueezeNet includes a generator network and a discriminator network.

[0027] Compared with the existing technology, the beneficial effects achieved by the present invention are as follows: the present invention outputs a defogging fog-free image by inputting the collected foggy image into a generative adversarial network model based on SqueezeNet; wherein, the generative adversarial network model based on SqueezeNet includes a generator network and a discriminator network; it has the advantages of good image defogging effect, less loss of image details after defogging, and the model training process does not depend on the noise distribution of the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the main process of a SqueezeNet-based generative adversarial network image dehazing method provided by an embodiment of the present invention;

[0029] Figure 2 2 is a schematic diagram of the structure of the feature extraction network SqueezeNet in an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of the internal structure of the Fire module in an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of the structure of the attention block in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0033] Example 1:

[0034] like Figures 1 to 4 As shown, a SqueezeNet-based generative adversarial network image defogging method includes: collecting foggy images; inputting the collected foggy images into a constructed SqueezeNet-based generative adversarial network model, and outputting a defogged fog-free image; wherein the SqueezeNet-based generative adversarial network model includes a generator network and a discriminator network.

[0035] Step 1: Obtain the outdoor training set OTS in the RESIDE dataset as the original dataset for training the neural network, and divide it into training samples, test samples, and validation samples.

[0036] Step 2: Take the foggy image in the training sample as input and design a SqueezeNet-based generative adversarial network model. The SqueezeNet-based generative adversarial network model includes a generator network and a discriminator network.

[0037] The generator network includes a feature extraction network that integrates attention blocks and an image reconstruction network that are sequentially connected. The discriminator network includes multiple convolutional layers, fully connected layers, and sigmoid activation layers that are sequentially connected. The feature extraction network SqueezeNet contains a multi-layer neural network structure, which is: data input layer, convolutional layer, maximum pooling layer, 3 Fire modules, maximum pooling layer, 4 Fire modules, maximum pooling layer, Fire module, and convolutional layer. The data input layer is used to input foggy images. The input images are compressed and expanded by the Fire module. The output feature map resolution remains unchanged, but the number of channels changes. Figure 2 FIG. 1 is a schematic diagram of the structure of the feature extraction network SqueezeNet in an embodiment of the method of the present invention.

[0038] Among them, the core of the feature extraction network SqueezeNet is the Fire module. Figure 3Figure 2 shows the internal structure of the Fire module in an embodiment of the method of the present invention. The Fire module consists of two parts: the squeeze layer and the expand layer. The squeeze layer is a convolutional layer containing a 1×1 convolution kernel, and the expand layer is a convolutional layer containing 1×1 and 3×3 convolution kernels, both with a stride of 1. In the expand layer, the feature maps obtained by the 1×1 and 3×3 convolutions are concatenated, and each convolution layer uses the ReLu activation function. The Fire module has three adjustable parameters:

[0039] -s1: squeeze layer, the number of 1×1 convolution kernels, which is also the number of channels of the output feature map;

[0040] -e1: expand layer, the number of 1×1 convolution kernels, and also the number of channels of the output feature map;

[0041] -e3: expand layer, the number of 3×3 convolution kernels, which is also the number of channels of the output feature map;

[0042] The number of channels of the input feature map is M, and the number of channels of the final output feature map of the Fire module is e1+e3.

[0043] The feature extraction network SqueezeNet directly learns the fog density features from the blurred image, enabling the image reconstruction network to automatically adjust the weights of areas with different concentrations, achieving non-uniform processing of different concentrations of haze in real scenes.

[0044] The attention block has a two-branch structure, the upper branch uses two convolutional layers and two LeakyReLu layers; the lower branch uses an average pooling layer, a maximum pooling layer, a convolutional layer and a Leaky ReLu layer. Figure 4 The figure shows a schematic diagram of the attention block structure described in an embodiment of the method of the present invention. The upper branch obtains the pixel features of the input information after two 3×3 convolutions and Leaky ReLu processing; the lower branch adds the input information after average pooling and maximum pooling processing, and then performs 1×1 convolution and Leaky ReLu processing to obtain the channel features of the input information. The two consecutive convolutions and mapping processing of the upper branch strengthen the pixel features; the average pooling layer and the maximum pooling layer of the lower branch process the information, splice it, and then perform convolution and mapping processing, which can improve the feature extraction ability of the channel. After the double-branch processing, the obtained data is added pixel by pixel, and the Sigmoid function is used to configure the corresponding weight parameters for the channel and pixel. The image obtained after processing by the feature extraction network SqueezeNet is multiplied pixel by pixel with this weight parameter, thereby realizing pixel-by-pixel weighted processing of the original image, so that better processing effect can be obtained on non-uniformly distributed haze images.

[0045] The formula is as follows:

[0046] M(F)=σ((M P (F)+M C (F)) (1)

[0047] Among them, F is the input feature map obtained from the training sample set, M(F) is the final weight parameter, and M P (F) is the pixel weight parameter, M C (F) is the channel weight parameter, and σ is the Sigmoid activation function.

[0048] The image reconstruction network adjusts the feature maps output by the feature extraction network to the same size through deconvolution, then connects them into a feature map, and reconstructs the image through deconvolution, activation and element-wise addition fusion.

[0049] Step 3: Construct the loss function.

[0050] Added perceptual loss function for:

[0051]

[0052] Among them, j represents the jth layer of the network, CjHjWj represents the size of the feature map of the jth layer in the network, represents the output feature of the jth layer of the real fog-free image, Represents the output features of the jth layer of the fog-free image estimated by the network;

[0053] Discriminator loss function L1:

[0054]

[0055] Among them, D(x i ) represents the discriminant result of the discriminator for the i-th image in the training sample set, D(G(z i )) represents the discriminator's judgment result on the i-th generated image generated by the generator, m is the number of training samples; the generator loss function L2:

[0056]

[0057] Among them, D(G(z i )) represents the discrimination result of the discriminator for the i-th generated image generated by the generator, and m is the number of training samples;

[0058] When the generator achieves its best performance, the discriminator cannot determine whether the image is a dehazed image or a haze-free image, and the discriminator's loss function is 0.5. In this state, the generator can produce results that are closest to the real haze-free image.

[0059] Step 4: Model training. Train a generative adversarial network based on SqueezeNet. The goal of network training is to minimize the discriminator loss function in Step 3 by continuously adjusting the weight parameters to 0.5, the generator loss function to 0, and all other loss functions. First, initialize the weight parameters of each stage of the network using a random Gaussian distribution with a mean of 0 and a standard deviation of 0.001.

[0060] After initialization, the stochastic gradient descent algorithm is used to update the weight parameters, and the update rule follows formula (5):

[0061]

[0062] Among them, α is the learning rate, W is the weight parameter after update, and W0 is the weight parameter before update. Calculated by the back-propagation algorithm.

[0063] Among them, the main steps of the backpropagation algorithm are: first, forward propagation of the given sample is performed to obtain the final output value of the network; then, the total error is calculated, and the partial derivative of the total error is used to obtain the influence of a certain node on the final output.

[0064] Therefore, the complete network training model is as follows:

[0065] Initialize the weight parameters of the network at each stage;

[0066] For each sample i, first use forward propagation to calculate the total error, then use backpropagation to calculate the partial derivatives of each weight parameter, and finally update the weight parameters according to the above formula. Repeat this step until the discriminator loss function is 0.5 and the other loss functions are minimized.

[0067] Step 5: Input the foggy image into the generator of the trained SqueezeNet-based generative adversarial network model, and the output result is the defogging fog-free image.

[0068] This paper uses a generative adversarial network based on SqueezeNet to perform image dehazing. A foggy image is fed into a pretrained network to directly produce a fog-free image. Because the SqueezeNet network extracts fog density features, the dehazing network automatically adjusts the weights of regions of varying concentrations, more effectively separating haze from image content and achieving non-uniform treatment of varying haze concentrations in real-world scenarios. An improved attention block enables the dehazing network to extract key image features, improving its performance. The introduction of a perceptual loss helps recover more image detail, avoiding the loss of spatial detail often associated with traditional convolutional neural networks.

[0069] Example 2:

[0070] Based on the SqueezeNet-based generative adversarial network image defogging method described in Example 1, this embodiment provides a SqueezeNet-based generative adversarial network image defogging device, including: an image acquisition module for acquiring foggy images; a defogging module for inputting the acquired foggy images into a constructed SqueezeNet-based generative adversarial network model, and outputting a defogged fog-free image; wherein the SqueezeNet-based generative adversarial network model includes a generator network and a discriminator network.

[0071] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image defogging method, characterized in that: include: Collect foggy images; The collected foggy image is input into the constructed SqueezeNet-based generative adversarial network model, and the defogging fog-free image is output; The SqueezeNet-based generative adversarial network model includes a generator network and a discriminator network. The generator network includes a sequentially connected feature extraction network SqueezeNet and an image reconstruction network; the discriminator network includes a plurality of sequentially connected convolutional layers, fully connected layers, and sigmoid activation layers; The feature extraction network SqueezeNet learns fog density features from foggy images, which are used in the image reconstruction network to automatically adjust the weights of areas with different concentrations, thereby achieving non-uniform processing of different concentrations of haze in real scenes. The feature extraction network SqueezeNet includes a data input layer, a convolutional layer, a maximum pooling layer, three Fire modules, a maximum pooling layer, four Fire modules, a maximum pooling layer, a Fire module, and a convolutional layer. The data input layer is used to receive the input foggy image, and the input foggy image is compressed and expanded by the Fire module. The output feature map resolution remains unchanged, but the number of channels changes; Each Fire module includes a squeeze layer and an expand layer; the squeeze layer is a convolution layer containing a 1×1 convolution kernel, and the expand layer is a convolution layer containing 1×1 and 3×3 convolution kernels; in the expand layer, the feature maps obtained by the 1×1 and 3×3 convolutions are connected, and each convolution layer uses a ReLu activation function.

2. The image defogging method according to claim 1, characterized in that: The feature extraction network SqueezeNet integrates an attention block, which includes an upper branch and a lower branch. Among them, the upper branch uses two convolutional layers and two LeakyReLU layers to obtain the pixel features of the input information; The lower branch uses an average pooling layer, a maximum pooling layer, a convolutional layer, and a LeakyReLU layer to obtain the channel features of the input information; The data obtained by the upper branch and the lower branch are added pixel by pixel, and the Sigmoid function is used to configure the corresponding weight parameters for the channel and pixel. Then, the image obtained after processing by the feature extraction network SqueezeNet is multiplied pixel by pixel using formula (1) to achieve pixel-by-pixel weighted processing of the foggy image, thereby obtaining a better processing effect on the non-uniformly distributed foggy image. （1） in, F is the input feature map obtained from the training sample set, is the final weight parameter, is the pixel weight parameter, is the channel weight parameter, is the sigmoid activation function.

3. The image defogging method according to claim 2, characterized in that: The image reconstruction network adjusts the feature maps output by the feature extraction network to a consistent size through deconvolution, then connects them into a feature map, and reconstructs the image through deconvolution, activation and element-wise addition fusion.

4. The image defogging method according to claim 3, characterized in that: The loss function in the SqueezeNet-based generative adversarial network model includes: Introduced perceptual loss function for: （2） in, j Represents the network j layer, Indicates the network j The size of the feature map of the layer, Represents the real fog-free image j The output features of the layer, Represents the haze-free image estimated by the network j Output features of the layer; Discriminator loss function : （3） in, Represents the discriminant result of the discriminator for the i-th image in the training sample set, represents the discriminator's discrimination result for the i-th generated image generated by the generator, and m is the number of training samples; Generator loss function : （4） in, represents the discrimination result of the discriminator for the i-th generated image generated by the generator, and m is the number of training samples.

5. The image defogging method according to claim 4, characterized in that: The training method of the SqueezeNet-based generative adversarial network model includes: Initialize the weight parameters of the network at each stage; For each sample i , first use forward propagation to calculate the total error, then use back propagation to calculate the partial derivatives of each weight parameter, and finally update the weight parameters according to formula (5); repeat this step until the result of each loss function reaches the set value, （5） in, is the learning rate, is the updated weight parameter, is the weight parameter before updating, Calculated by the back-propagation algorithm.

6. An image defogging device, characterized in that: include: Image acquisition module, used to collect foggy images; The defogging module is used to input the collected foggy image into the constructed SqueezeNet-based generative adversarial network model and output the defogging fog-free image; The SqueezeNet-based generative adversarial network model includes a generator network and a discriminator network. The generator network includes a sequentially connected feature extraction network SqueezeNet and an image reconstruction network; the discriminator network includes a plurality of sequentially connected convolutional layers, fully connected layers, and sigmoid activation layers; The feature extraction network SqueezeNet learns fog density features from foggy images, which are used in the image reconstruction network to automatically adjust the weights of areas with different concentrations, thereby achieving non-uniform processing of different concentrations of haze in real scenes. The feature extraction network SqueezeNet includes a data input layer, a convolutional layer, a maximum pooling layer, three Fire modules, a maximum pooling layer, four Fire modules, a maximum pooling layer, a Fire module, and a convolutional layer. The data input layer is used to receive the input foggy image, and the input foggy image is compressed and expanded by the Fire module. The output feature map resolution remains unchanged, but the number of channels changes; Each Fire module includes a squeeze layer and an expand layer; the squeeze layer is a convolution layer containing a 1×1 convolution kernel, and the expand layer is a convolution layer containing 1×1 and 3×3 convolution kernels; in the expand layer, the feature maps obtained by the 1×1 and 3×3 convolutions are connected, and each convolution layer uses a ReLu activation function.

Images