An image dehazing system based on dual discriminator heterogeneous CycleGAN

Abstract

An image defogging system based on dual discriminator heterogeneous CycleGAN comprises a first image defogging training module and a second image defogging training module. The first image defogging training module comprises a first image input layer, a defogging generator G connected in sequence, BN , restore fog generator #imgabs0# Among them, the first image input layer is given to the defogging generator G BN Input foggy image X, dehazing generator G BN Input the fog-free image G to the restored fog generator #imgabs1# BN (X), restore fog generator #imgabs2 #output foggy image #imgabs3 #The second image defogging training module includes a second image input layer, a restored fog generator F connected in sequence BN , Dehazing Generator #imgabs4#The purpose of this invention is to solve the learning confusion problem between generators of the same task caused by directly or indirectly using the CycleGAN method for image dehazing.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an image defogging system based on dual discriminator heterogeneous CycleGAN. Background Art

[0002] Due to the influence of tiny particles in the weather such as haze, the visible light system will have problems such as image blur, color distortion, and contrast reduction during the imaging process, which seriously affects the image quality. In advanced visual tasks such as target detection, face recognition alignment, and image classification, image dehazing is usually used as a preprocessing process for high-level tasks to restore high-quality image content information for later processing. Therefore, in the past few years, image dehazing has received extensive research attention in the field of vision.

[0003] In the early defogging algorithms, most methods estimated intermediate parameters such as transmission maps and global atmospheric light values ​​through prior information, and then used the atmospheric scattering model to achieve image defogging. Although the prior-based methods have made significant progress, these methods usually make inaccurate estimates under unconstrained conditions, resulting in the quality of the restored fog-free images being less than ideal. With the development of deep learning, researchers have proposed many defogging methods based on convolutional neural networks, which are mainly divided into defogging methods based on intermediate parameter estimation and end-to-end defogging methods. Although the methods based on intermediate parameter estimation have made certain progress compared with the prior-based methods, they are prone to introduce color distortion and artifacts when the parameter estimation is inaccurate. The end-to-end defogging method does not need to estimate the intermediate parameters of the atmospheric scattering model, but most end-to-end methods require paired data sets when training the model. It is well known that it is difficult to collect large-scale real foggy and fog-free paired data sets. Most of the data sets used to train the model use artificially synthesized fog images. However, there is a certain gap between artificially synthesized foggy images and the distribution of fog in real scenes.

[0004] In recent years, with the introduction of the adversarial generative network (GAN) and the development of various GAN network variants, research based on the GAN network idea has demonstrated the powerful capabilities of the GAN network in image generation fields such as image super-resolution and image enhancement. In particular, the introduction of the CycleGAN network framework, which can achieve image style transfer using unpaired datasets, has attracted a lot of attention from researchers in the field of computer vision. However, when using CycleGAN directly or methods based on CycleGAN for image dehazing, the following problems still exist:

[0005] (1) In the defogging framework based on CycleGAN, in order to realize the unpaired dataset training model, it is necessary to train two generators: defogging and restoration of fog. Taking the defogging generator as an example, the input data of the generator includes the original foggy image and the image generated by the restoration of fog generator. Obviously, these two different distributions of data will affect the convergence speed and effect of the network;

[0006] (2) Most dehazing algorithms based on GAN networks lack attention to local areas of the image, resulting in the quality of the restored dehazed image in local areas not being satisfactory;

[0007] (3) The channel attention commonly used in dehazing networks is a first-order statistic that does not consider the relationship between feature channels, lacks effective geometric constraints, and cannot fully utilize the expressive power of features. Summary of the invention

[0008] The purpose of the present invention is to solve the problem of learning confusion between generators with the same task in the process of directly or indirectly using the CycleGAN method for image defogging; in addition, since these methods lack exploration of local areas when restoring images, only first-order channel attention is used in the constructed network structure, ignoring the effective use of deep-level channel-related information. Therefore, the purpose of the present invention is to increase the network's attention to local areas and fully explore channel-related information that is crucial for image defogging from different angles.

[0009] An image defogging system based on dual discriminator heterogeneous CycleGAN comprises a first image defogging training module and a second image defogging training module. The first image defogging training module comprises a first image input layer, a defogging generator G connected in sequence, BN , restore fog generator

[0010] Among them, the first image input layer gives the defogging generator GBN an input of the foggy image X, and the defogging generator GBN gives the restoration fog generator Input haze-free image G BN (X) Restore the fog generator Output foggy image

[0011] The second image defogging training module includes a second image input layer, a recovery fog generator F connected in sequence BN , Dehaze Generator

[0012] Among them, the second image input layer is given to the restoration fog generator F BN Input fog-free image Y, restore fog generator F BN Dehaze Generator Input foggy image FBN (Y), Dehaze Generator Output fog-free image

[0013] Dehaze Generator G BN With Dehaze Generator Between, and restoring fog generator F BN With restored fog generator The parameters of each convolution kernel layer are shared between the generators of the same task, that is, the parameters of the convolution layer are shared between the generators of the same task; and each generator has its own independent BatchNorm layer, such as Figure 1 The dotted line in . The BatchNorm operation forces the input data of each layer in the generator to return to their respective standard normal distribution, while the independent BatchNorm layer enables the generator to independently learn the data distribution in the input generator. BN Learn the distribution of foggy images X. Dehazing Generator Learning the foggy image F BN (Y) Distribution. Restore the fog generator F BN Learn the distribution of the haze-free image Y and restore the haze generator Learning haze-free images G BN (X) distribution.

[0014] It also includes a global discriminator Global Discriminator

[0015] Global Discriminator Used to distinguish foggy image X, foggy image and the foggy image F BN (Y); Global Discriminator Used to distinguish between fog-free image Y and fog-free image and the haze-free image G BN (X).

[0016] It also includes a local discriminator Local Discriminator Local Discriminator Local Discriminator The input is the global discriminator Global Discriminator The image is cropped from

[0017] Local Discriminator Used to identify foggy images X and generate foggy images cyclically And generate foggy image FBN(Y); local discriminator Used to distinguish fog-free image Y, fog-free image A fixed-size image cropped from the haze-free image GBN(X);

[0018] The model also adds the cycle consistency loss and the perceptual loss. The input of the cycle consistency loss and the perceptual loss is the foggy image X and As well as the fog-free image Y and the fog-free image The purpose of the loss function is to constrain the generator, and under the constraints of the loss function, the network model begins to train and gradually achieves the effect of image dehazing.

[0019] An image dehazing generator comprises an image input layer, a plurality of convolutional unit layers, a pooling layer, a convolutional layer, a multi-order channel attention module, a plurality of upsampling layers, and an image output layer connected in sequence;

[0020] The image input layer inputs the foggy image, and the image output layer outputs the fog-free image.

[0021] The multiple convolutional layers include a first convolutional unit layer, a second convolutional unit layer, a third convolutional unit layer, a fourth convolutional unit layer, a fifth convolutional unit layer, and a sixth convolutional unit layer that are connected in sequence, and each convolutional unit layer includes a convolution layer, an activation layer, and a batch normalization layer.

[0022] The multi-order channel attention module includes an upper part and a lower part. The upper part is a first-order channel attention module, which is composed of an upper input layer, an average pooling layer, a convolution layer, an activation layer, and an upper output layer from input to output. The data output by the upper output layer is multiplied by the data input by the upper input layer to obtain the first output result A.

[0023] The lower part is the second-order channel attention module, which is the lower input layer, reshape module, covariance normalization module, covariance pooling module, convolution layer, activation layer, and lower output layer from input to output. The data output by the lower output layer is multiplied by the data input by the lower input layer to obtain the second output result B.

[0024] The first output result A and the second output result B are added together to obtain a characteristic graph of multi-order channel calibration;

[0025] Among them, the upper input layer and the lower input layer input the same data.

[0026] When it is used, the following steps are taken:

[0027] Step 1: Start by feeding the feature map X into the multi-stage channel attention module;

[0028] Step 2: Perform first-order channel attention weighting on the feature map X;

[0029] Step 3: Perform second-order channel attention weighting on feature X;

[0030] Step 4: Fuse the weighted feature map obtained in step 2 with the weighted feature map obtained in step 3;

[0031] Step 5: Output the fused image and end;

[0032] In step 2, when performing first-order channel attention weighting, the following steps are taken:

[0033] Step 2-1) The global spatial information of the channel obtained by global average pooling of the input X;

[0034] Step 2-2) The obtained global spatial information is convolved once and passed through a RELU activation function;

[0035] Step 2-3) Perform another convolution operation on the output of step 2-2) and pass it through the Sigmoid activation function;

[0036] Step 2-4) Multiply the output of step 2-3) by X and output it.

[0037] In step 3, when performing the second-order channel attention weighting, the following steps are taken:

[0038] Step 3-1) Reshape the input X and expand it according to the channel dimension;

[0039] Step 3-2) Perform covariance normalization on the reshaped feature map;

[0040] Step 3-3) Perform covariance pooling operation on the feature map after covariance normalization;

[0041] Step 3-4) Perform convolution operation on the feature map after covariance pooling and pass it through RELU activation function;

[0042] Step 3-5) The feature map obtained in step 3-4) is convolved again and passed through the Sigmoid activation function;

[0043] Step 3-6) multiplies the output of step 3-5) by X and outputs it.

[0044] Compared with the prior art, the present invention has the following technical effects:

[0045] 1) The outdoor image defogging model provided by the present invention does not require a paired data set to train the network. In addition, in the defogging model system framework of the present invention, generators with the same task can independently learn data with different distributions. In addition, the dual discriminator mechanism in the system not only constrains the generator from a global perspective, but also increases the network's attention to local areas.

[0046] 2) The multi-order channel attention module in the present invention fuses the recalibrated feature maps obtained from different statistics, fully explores the interdependence between feature channels, and performs more accurate multi-order channel recalibration, thereby improving the visual quality of the dehazed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0048] Figure 1 It is the overall network structure diagram of the present invention;

[0049] Figure 2 This is the generator structure diagram;

[0050] Figure 3 Structural diagram of the multi-stage channel attention module. DETAILED DESCRIPTION

[0051] The present invention provides an image defogging system based on a dual discriminator heterogeneous CycleGAN, which comprises a first image defogging training module and a second image defogging training module. The first image defogging training module comprises a first image input layer, a defogging generator G connected in sequence, BN , restore fog generator

[0052] Among them, the first image input layer is given to the dehazing generator G BN Input foggy image X, dehazing generator G BN Restore fog generator Input haze-free image G BN (X) Restore the fog generator Output foggy image

[0053] The second image defogging training module includes a second image input layer, a recovery fog generator F connected in sequence BN , Dehaze Generator

[0054] Among them, the second image input layer is given to the restoration fog generator F BN Input fog-free image Y, restore fog generator F BN Dehaze Generator Input foggy image F BN (Y), Dehaze Generator Output fog-free image

[0055] Dehaze Generator G BN With Dehaze Generator Between, and restoring fog generator FBN and restoring fog generator The parameters of each convolution kernel layer are shared between the generators of the same task, that is, the parameters of the convolution layer are shared between the generators of the same task; and each generator has its own independent BatchNorm layer, such as Figure 1 The dotted line in . The BatchNorm operation forces the input data of each layer in the generator to return to their respective standard normal distribution, while the independent BatchNorm layer enables the generator to independently learn the data distribution in the input generator. BN Learn the distribution of foggy images X. Dehazing Generator Learning the foggy image F BN (Y) Distribution. Restore the fog generator F BN Learn the distribution of the haze-free image Y and restore the haze generator Learning haze-free images G BN (X) distribution.

[0056] It also includes a global discriminator Global Discriminator

[0057] Global Discriminator Used to distinguish foggy image X, foggy image and the foggy image F BN (Y); Global Discriminator Used to distinguish between fog-free image Y and fog-free image and the haze-free image G BN (X).

[0058] It also includes a local discriminator Local Discriminator Local Discriminator Local Discriminator The input is the global discriminator Global Discriminator The image is cropped from

[0059] Local Discriminator Used to identify foggy images X and generate foggy images cyclically and generate a foggy image F BN (Y); Local Discriminator Used to distinguish fog-free image Y, fog-free image and the haze-free image G BN (X) Cropped fixed-size image;

[0060] The model also adds the cycle consistency loss and the perceptual loss. The input of the cycle consistency loss and the perceptual loss is the foggy image X and As well as the fog-free image Y and the fog-free image The purpose of the loss function is to constrain the generator, and under the constraints of the loss function, the network model begins to train and gradually achieves the effect of image dehazing.

[0061] For the image defogging system based on dual discriminator heterogeneous CycleGAN, the following embodiments are further provided:

[0062] like Figure 1 As shown, Figure 1 The overall system of training image defogging network is given. Since the present invention does not require paired data sets to train the network, the foggy image X and the clear fog-free image Y input to the network are in a non-paired relationship. The foggy image X is passed through the defogging generator G BN Generate a haze-free image G BN (X), and then the haze-free image G BN (X) Input recovery fog generator In the example above, we get a foggy image. Similarly, the haze-free image Y is restored by the haze generator F BN Generate foggy image F BN (Y), and then the foggy image F BN (Y) sent to the defogging generator In the image, we get a fog-free image.

[0063] In order to make the generators of the same task learn the distribution of different data independently, this paper combines the dehazing generator GBN with In between, restore the fog generator F BN and All parameters of the convolutional layer, activation layer, and pooling layer are shared between them, and they have independent Batchnorm layers, such as Figure 1 In addition, in order to increase the network's attention to local areas, the proposed system contains two types of visual field discriminators, namely the global discriminator and local discriminator

[0064] The global discriminator has 8 convolution operations. The first 7 convolution operations are followed by the activation function LeakyReLU and batch normalization BatchNorm. Except for the first and last convolution operations, the remaining convolution operations will reduce the size of the feature map. Global discriminator Used to distinguish foggy image X, foggy image and the foggy image F BN (Y). Global Discriminator Used to distinguish between fog-free image Y and fog-free image and the haze-free image G BN (X).

[0065] Different from the global discriminator, the local discriminator has 6 convolution operations, and the input of the local discriminator is randomly cropped from the image input to the global discriminator. For example, Used to distinguish foggy images X and foggy images and the foggy image F BN (Y) Five cropped images of size 32*32. In this way, the network's attention to local areas is increased. If only the global discriminator is used, the discriminator may misjudge a foggy image with uneven fog density distribution as a fog-free image. The field of view of the local discriminator is a 32*32 patch in the image. Compared with using only the global discriminator, the local discriminator can enhance the network's adaptability to local foggy areas. The experimental results show that under the action of the global-local discriminator, the fog-free image restored by the present invention has a better visual effect.

[0066] When the system using the image dehazing network trains the dehazing generator, the following steps are taken:

[0067] Step (1) sends the foggy image in the network model training set 1 to the defogging generator 2, and at the same time sends the fog-free image in the network model training set to the restoration fog generator;

[0068] Step (2) sends the output of step (1) to the recovery mist generator 3 and the demisting generator 2 respectively.

[0069] Step (3) The foggy image and fog-free image in the network model training set 1 and the output in 2 are respectively sent to the loss function module 6 to construct the cycle consistency loss and the perceptual loss;

[0070] Step (4) sends the foggy image and the fog-free image in the network model training set 1, the output in step (1), and the output in step (2) to the corresponding global discriminator 4 respectively to distinguish true from false;

[0071] Step (5) randomly crops the foggy image and fog-free image in the network model training set 1, the output in step (1), and the output in step (2) into a plurality of images, and then respectively sends them to the corresponding local discriminator 5 for true and false discrimination;

[0072] Finally, the dehazing generator in the trained model is taken out separately, which can restore the input foggy image to a fog-free image.

[0073] The present invention also provides an image defogging generator, which includes an image input layer, a plurality of convolutional unit layers, a pooling layer, a convolutional layer, a multi-order channel attention module, a plurality of upsampling layers, and an image output layer connected in sequence;

[0074] The image input layer inputs the foggy image, and the image output layer outputs the fog-free image.

[0075] Among them, Figure 2 , as shown in Figure 3. The multi-order channel attention module can fully explore the correlation between channels and adaptively recalibrate the channel feature map.

[0076] The multiple convolutional layers include a first convolutional unit layer, a second convolutional unit layer, a third convolutional unit layer, a fourth convolutional unit layer, a fifth convolutional unit layer, and a sixth convolutional unit layer that are connected in sequence, and each convolutional unit layer includes a convolution layer, an activation layer, and a batch normalization layer.

[0077] The multi-order channel attention module includes an upper part and a lower part. The upper part is a first-order channel attention module, which is composed of an upper input layer, an average pooling layer, a convolution layer, an activation layer, and an upper output layer from input to output. The data output by the upper output layer is multiplied by the data input by the upper input layer to obtain the first output result A.

[0078] The lower part is the second-order channel attention module, which is the lower input layer, reshape module, covariance normalization module, covariance pooling module, convolution layer, activation layer, and lower output layer from input to output. The data output by the lower output layer is multiplied by the data input by the lower input layer to obtain the second output result B.

[0079] The first output result A and the second output result B are added together to obtain a characteristic graph of multi-order channel calibration;

[0080] Among them, the upper input layer and the lower input layer input the same data.

[0081] When it is used, the following steps are taken:

[0082] Step 1: Start by feeding the feature map X into the multi-stage channel attention module;

[0083] Step 2: Perform first-order channel attention weighting on the feature map X;

[0084] Step 3: Perform second-order channel attention weighting on feature X;

[0085] Step 4: Fuse the weighted feature map obtained in step 2 with the weighted feature map obtained in step 3;

[0086] Step 5: Output the fused image and end;

[0087] In step 2, when performing first-order channel attention weighting, the following steps are taken:

[0088] Step 2-1) The global spatial information of the channel obtained by global average pooling of the input X;

[0089] Step 2-2) The obtained global spatial information is convolved once and passed through a RELU activation function;

[0090] Step 2-3) Perform another convolution operation on the output of step 2-2) and pass it through the Sigmoid activation function;

[0091] Step 2-4) Multiply the output of step 2-3) by X and output it.

[0092] In step 3, when performing the second-order channel attention weighting, the following steps are taken:

[0093] Step 3-1) Reshape the input X and expand it according to the channel dimension;

[0094] Step 3-2) Perform covariance normalization on the reshaped feature map;

[0095] Step 3-3) Perform covariance pooling operation on the feature map after covariance normalization;

[0096] Step 3-4) Perform convolution operation on the feature map after covariance pooling and pass it through RELU activation function;

[0097] Step 3-5) The feature map obtained in step 3-4) is convolved again and passed through the Sigmoid activation function;

[0098] Step 3-6) multiplies the output of step 3-5) by X and outputs it.

[0099] For the image defogging generator, further, the following embodiments are provided:

[0100] like Figure 2 As shown, when applying the image defogging network of the present invention, only the defogging generator GBN , feed the foggy image X into the generator G BN In the process, it first undergoes 6 convolution, activation, and batch normalization operations, and then performs an adaptive pooling layer, which can adapt the generator to input images of any size. Then it performs one more convolution, activation, and batch normalization operation, and then sends the output to the multi-order channel attention module (mcc). Finally, it outputs a clear and fog-free image through multiple upsampling. The effectiveness of the invention is verified on three public synthetic data sets and real foggy data sets, and excellent results are achieved in both objective evaluation indicators (PSNR, SSIM) and subjective visual effects.

[0101] Figure 3 The structure of the multi-stage channel attention module is shown, such as Figure 3 As shown in the figure, the upper part is the first-order channel attention module, which performs average pooling, convolution and activation operations on the input to obtain a first-order channel calibrated image. The lower part is the second-order channel attention module, which reshapes the input into a two-dimensional feature map and then performs covariance normalization. Covariance normalization plays an important role in representing the differences between variables. Then covariance pooling is performed, and finally the covariance pooled image is convolved and activated to obtain weights that can characterize the relationship between channels, and the obtained weights are multiplied by the input. Finally, the output of the first-order channel attention module and the output of the second-order channel attention module are fused to obtain a multi-order channel calibrated feature map.

Claims

1. An image dehazing system based on dual discriminator heterogeneous CycleGAN, characterized in that: The method comprises a first image defogging training module and a second image defogging training module. The first image defogging training module comprises a first image input layer, a defogging generator G connected in sequence. BN , restore fog generator Among them, the first image input layer is given to the dehazing generator G BN Input foggy image X, dehazing generator G BN Restore fog generator Input haze-free image G BN (X) Restore the fog generator Output foggy image The second image defogging training module includes a second image input layer, a recovery fog generator F connected in sequence BN , Dehaze Generator Among them, the second image input layer is given to the restoration fog generator F BN Input fog-free image Y, restore fog generator F BN Dehaze Generator Input foggy image F BN (Y), Dehaze Generator Output fog-free image Also includes a global discriminator Global Discriminator Global Discriminator Used to distinguish foggy image X, foggy image and the foggy image F BN (Y); Global Discriminator Used to distinguish between fog-free image Y and fog-free image and the haze-free image G BN (X); It also includes a local discriminator Local Discriminator Local Discriminator Local Discriminator The input is the global discriminator Global Discriminator The image is cropped from Local Discriminator Used to distinguish foggy images X and foggy images and the foggy image F BN (Y) Cropped fixed-size image; local discriminator Used to distinguish fog-free image Y, fog-free image and the haze-free image G BN (X) Cropped fixed-size image; The model also adds cycle consistency loss and perceptual loss; the input of cycle consistency loss and perceptual loss is the foggy image X and As well as the fog-free image Y and the fog-free image 2. The image defogging system based on dual discriminator heterogeneous CycleGAN according to claim 1, characterized in that: Dehaze Generator G BN With Dehaze Generator Between, and restoring fog generator F BN With restored fog generator The parameters in each layer of convolution kernel are shared between the generators of the same task, that is, the parameters of the convolution layer are shared between the generators; and each generator has its own independent BatchNorm layer.

Images