Multi-depth fusion image defogging method based on conditional generative adversarial network

Abstract

The invention discloses a multi-depth fusion image defogging method based on a conditional generative adversarial network, and the method comprises the steps: obtaining a data set containing foggy images and clear images in one-to-one correspondence with the foggy images, and dividing the data set into a training set and a test set; a multi-depth fusion image defogging model is constructed based on the framework of the conditional generative adversarial network, the multi-depth fusion image defogging model comprises a generator and a discriminator, the generator is used for generating a foggy image, the generator is used for analyzing whether an input image is a real image or a generated image, and feedback is provided for the generator according to an output result; training the constructed multi-depth fusion image defogging model by using the training set to obtain a trained image defogging model; and testing the test set by using the trained image defogging model to obtain a defogged image. According to the method, the concentration and the size of the haze can be sensed more accurately, and the problem that the overall defogging effect is poor due to the fact that the haze distribution concentration is not uniform under the real haze condition in the prior art is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image defogging, and in particular to an image defogging method based on multi-depth fusion of a conditional generative adversarial network (CGAN). Background Art

[0002] Haze weather not only brings inconvenience to human production activities, but also affects the atmospheric environment and thus has a negative impact on human health. For fog, an atmospheric phenomenon, due to the presence of a large number of particles that can absorb and scatter light, visibility is hindered, causing the acquired images to have problems such as reduced visibility, reduced contrast and color attenuation, resulting in serious image degradation and loss of image information. In daily life, traffic supervision and security monitoring cannot be separated from high-quality images. Therefore, it is of great significance to study how to more efficiently remove fog from images in real scenes.

[0003] People at home and abroad have already started to study image dehazing. According to different methods, image dehazing methods can be divided into two categories: traditional prior methods and deep learning-based methods. For example:

[0004] Reference [1]: HE K, SUN J, TANG X. Single image haze removal using dark channel prior [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353. The dark channel prior method is described in [1].

[0005] Reference [2]: Qingsong Z, Jiaming M, Ling SA Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior. [J]. IEEE transactions on image processing: a publication of the IEEE Signal Processing Society, 2015, 24(11): 3522-33. The color attenuation prior method is described in the paper.

[0006] These methods can use the characteristics of the physical model to roughly estimate the potential clear image, but in complex real scenes, this prior information is often invalid and the calculation is complex. The latter mostly adopts a symmetrical encoder and decoder structure, among which the most common network structure is U-Net. The encoder and decoder respectively play the role of extracting multi-scale features of the image and gradually restoring the clear image. However, based on this type of method, too little processing is done on the features of each scale of the foggy image, which easily leads to deficiencies in detail recovery.

[0007] Some existing dehazing models ignore the fact that the concentration of haze in different areas of the image is different. For areas with dense haze, the dehazing conversion may be insufficient, while for areas with light haze or less haze, excessive conversion may occur, resulting in color cast, blur and other problems. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides an image dehazing method based on multi-depth fusion of conditional generative adversarial network (CGAN). The method can more accurately sense the concentration and size of haze, and effectively solves the problem of poor global dehazing effect caused by uneven haze distribution concentration in real haze situations faced by previous methods; and can retain the original details and colors as much as possible while dehazing.

[0009] The technical solution adopted by the present invention is:

[0010] The multi-depth fusion image dehazing method based on conditional generative adversarial network includes the following steps:

[0011] Step 1: Obtain a data set containing foggy images and clear images corresponding to the foggy images, and divide the data set into a training set and a test set;

[0012] Step 2: Based on the framework of conditional generative adversarial network, a multi-depth fusion image defogging model is constructed. The multi-depth fusion image defogging model includes a generator and a discriminator. The generator is used to generate a foggy image. The generator is used to analyze whether the input image is a real image or a generated image, and provide feedback to the generator according to the output result;

[0013] Step 3: Use the training set to train the multi-depth fusion image defogging model constructed in step 2 to obtain a trained image defogging model;

[0014] Step 4: Use the image dehazing model trained in step 3 to test the test set and obtain the dehazed image.

[0015] In the step 1, a foggy image dataset for training is constructed. First, 13,990 indoor foggy images generated from 1,399 clear images and 8,970 outdoor foggy images generated from 897 clear images are selected from the open source real single image defogging dataset RESIDE as the original data source; then, they are divided into a training set and a test set in a ratio of 8:2.

[0016] In the step 2, a multi-deep fusion adversarial generation network (MDF-GAN) for image defogging is constructed, which specifically includes:

[0017] Build a multi-depth fusion generator for generating close-to-real images. The generator can learn the mapping from foggy images to their corresponding real fog-free images. The generator includes a Haze Learning Block (HLB) that can extract the features of foggy images, a Multi-depth Projection Module (MDPM) that fuses semantic features of different depths, an upsampling encoder, a downsampling decoder, and an enhancement module that can enhance the low-resolution high-semantic feature representation capability.

[0018] A discriminator is built to help the generator achieve the best fitting state. The output of the generator and its true haze-free image are provided as input to the discriminator for comparison. The discriminator then feeds back the quality judgment score to the generator.

[0019] Step 2 includes the following steps:

[0020] Step 2.1: Construct a haze learning unit for extracting features of foggy images;

[0021] Step 2.2: Construct a multi-depth projection module for fusing different depth semantic features;

[0022] Step 2.3: Build an enhancement module to enhance the low-resolution high-semantic feature representation capability;

[0023] Step 2.4: Build a multi-depth fusion generator for generating images close to the real image;

[0024] Step 2.5: Build a discriminator to help the generator achieve the best fitting state.

[0025] Step 2.1 includes the following steps:

[0026] Step 2.1.1: Build a global perception module GAM:

[0027] First, the feature map Input is processed with a Conv convolution with a convolution kernel size of 2×2, a stride of 1, and a padding of 1, a depthwise separable convolution with a convolution kernel size of 7×7, a stride of 3, and a group number of 3, and a batch normalization, and the output feature map F1∈R with unchanged resolution and number of channels is C×H×W , where C, H, and W represent the number of channels, height, and width, respectively; then, the feature map is input into the AdaptiveAvgPool2d global average pooling and AdaptiveMaxPool2d global maximum pooling with a length and width of 1×1 in parallel, and the results are concatenated to obtain the feature map T m ; Then, the feature map T m The weight value ω is obtained by inputting the convolution kernel size of 2×2, the step size of 1, and the padding of 1 into the Conv convolution and Sigmoid activation function in parallel; finally, the weight value ω is multiplied by the feature map F1 to obtain the feature map ωF1 with half the resolution and number of channels;

[0028] The output process of the output feature map ωF1 is expressed as:

[0029]

[0030] In the above formula, Conv(Input) means performing a convolution operation on the input feature map Input; Conv(Conv(Input)) means performing a convolution operation on the input feature map Input and then performing a convolution operation again; DsConv(Conv(Input)) means performing a convolution operation on the input feature map Input and then performing a depthwise separable convolution operation; BN(DsConv(Conv(Input))) means performing a convolution operation on the input feature map Input and then performing a depthwise separable convolution operation and finally performing a BatchNormalization operation; MaxPool(F1) means performing a global maximum pooling operation on the feature map F1; AvgPool(F1) means performing a global average pooling operation on the feature map F1; Conv(T m ) represents the feature map T m Perform convolution operation; Sigmoid(Conv(T m )) represents the feature map T m After the convolution operation, the Sigmoid activation function operation is performed;

[0031] Step 2.1.2: Build a dense module DenseNet consisting of four DenseBlocks. Each DenseBlock consists of a Conv convolution with a convolution kernel size of 2×2, a step size of 1, and a padding of 1, and a PReLU activation function. Each DenseBlock is densely connected. First, input the feature map ωF1 into the dense module DenseNet, and then perform a Conv convolution operation with a convolution kernel size of 1×1. Finally, perform a residual connection with the feature map ωF1, and output the feature map F2 with unchanged resolution and number of channels.

[0032] The feature map F2 can convey to the network whether a certain area in the image is important and the degree of defogging, so that the network can focus on the dense fog area. Therefore, the output process of the Haze Learning Block (HLB) is expressed as:

[0033]

[0034] In the above formula, H1 indicates that the feature map ωF1 is obtained after the first DenseBlock operation; H2 indicates that the feature map H1 is obtained after the second DenseBlock operation; H3 indicates that the feature map H2 is obtained after the third DenseBlock operation; H4 indicates that the feature map H3 is obtained after the fourth DenseBlock operation; Relu(Conv(ωF1)) indicates that the feature map ωF1 is convolved and then the Relu activation function is operated; PRelu(Conv(ωF1))) indicates that the feature map ωF1 is convolved and then the PRelu activation function is operated;

[0035] Conv(H1+ωF1) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed; Relu(Conv(H1+ωF1)) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed, and finally the Relu activation function operation is performed; PRelu(Conv(H1+ωF1))) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed, and finally the PRelu activation function operation is performed;

[0036] Conv(H2+H1+ωF1) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed; Relu(Conv(H2+H1+ωF1)) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed, and finally the Relu activation function operation is performed; PRelu(Conv(H2+H1+ωF1))) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed, and finally the PRelu activation function operation is performed;

[0037] Conv(H3+H2+H1+ωF1) means that residual operation is performed on feature map ωF1, feature map H1, feature map H2 and feature map H3, followed by convolution operation; Relu(Conv(H3+H2+H1+ωF1)) means that residual operation is performed on feature map ωF1, feature map H1, feature map H2 and feature map H3, followed by convolution operation and finally Relu activation function operation; PRelu(Conv(H3+H2+H1+ωF1))) means that residual operation is performed on feature map ωF1, feature map H1, feature map H2 and feature map H3, followed by convolution operation and finally PRelu activation function operation;

[0038] The haze learning unit HLB is used for defogging by region and degree. This unit can focus on the details in the haze image, determine the fog concentration in the captured haze image, locate the area in the image that needs to be converted, effectively defogging while avoiding over-defogging.

[0039] Step 2.2 includes the following steps:

[0040] S2.2.1: Build an Efficient Channel Attention (ECA) mechanism to integrate important features between non-adjacent layers:

[0041] First, the output feature map F2 of the haze learning unit HLB is globally averaged pooled using AdaptiveAvgPool2d; then, a one-dimensional convolution Conv1d operation with adaptive input size and nonlinear enhancement of the Sigmoid activation function are performed to obtain the channel dimension weight α; finally, the channel dimension weight α is multiplied by the feature map F2 to obtain the weighted screening map αF2 with unchanged resolution and number of channels. The output process of the weighted screening map αF2 is expressed as:

[0042]

[0043] In the above formula, AvgPool(F2) means that the feature map F2 is subjected to global average pooling operation; Conv1d(AvgPool(F2)) means that the feature map F2 is subjected to global average pooling operation and then subjected to one-dimensional convolution operation; Sigmoid(Conv1d(AvgPool(F2))) means that the feature map F2 is subjected to global average pooling operation and then subjected to one-dimensional convolution operation and finally subjected to Sigmoid activation function operation;

[0044] S2.2.2: Build a multi-depth projection module MDPM for fusing different depth semantic features:

[0045] First, the input feature map Input is subjected to a Conv convolution operation with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a nonlinear enhancement of the ReLU activation function; secondly, it is subtracted from the weight screening map αF2, and a ConvTranspose2d deconvolution operation with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a nonlinear enhancement of the ReLU activation function is performed together; then, it is added to the input feature map Input to obtain the output feature map J1 with unchanged resolution and number of channels. The output process of the output feature map J1 is expressed as:

[0046]

[0047] In the above formula, T1 represents the feature map obtained by performing convolution operation and Relu activation function operation on the input feature map Input and then subtracting it from the weight screening map αF2; Relu(Conv(Input)) represents performing convolution operation on the input feature map and then performing Relu activation function operation; T2 represents the feature map obtained by performing deconvolution operation and Relu activation function operation on the input feature map T1; ConvTranspose2d(T1) represents performing deconvolution operation on the input feature map T1; Relu(ConvTranspose2d(T1)) represents performing deconvolution operation on the input feature map T1 and then performing Relu activation function operation;

[0048] Feature maps at different levels carry different information and have different dehazing effects. However, the lack of information interaction between non-adjacent levels reduces the dehazing ability. Simply using the splicing method to fuse features of non-adjacent levels often has poor results. The multi-depth projection module MDPM can fuse semantic features of different depths and solve the above problem.

[0049] Step 2.3 includes the following steps:

[0050] S2.3.1: Build a residual group consisting of 3 residual blocks connected in series. Each residual block consists of a Conv convolution with a kernel size of 3×3 and a stride of 1, a ReLU activation function, a Conv convolution with a kernel size of 3×3 and a stride of 1, and a residual connection.

[0051] S2.3.2: Build the enhancement module. First, perform a ConvTranspose2d deconvolution operation on the input with a convolution kernel size of 4×4, a step size of 2, and a padding of 1. Then perform 6 residual group operations in sequence to obtain an enhanced feature map H with unchanged resolution and number of channels.

[0052] The enhancement process of the enhanced feature map H is expressed as:

[0053]

[0054] In the above formula, H θ It represents the feature map output after the input feature map Input passes through the first residual block; H 2θ Represents the input feature map H θ The feature map output after the second residual block; H 3θ Represents the input feature map H 2θ The feature map output after the third residual block; Relu(Conv(H θ )) represents the input feature map H θ After the convolution operation, the Relu activation function operation is performed; Conv(Relu(Conv(H θ ))) represents the input feature map H θ After the convolution operation, the Relu activation function operation is performed and the convolution operation is finally performed; Conv(H 2θ ) represents the input feature map H 2θ Perform convolution operation; Relu(Conv(H 2θ )) represents the input feature map H 2θ After the convolution operation, the Relu activation function operation is performed; Conv(Relu(Conv(H 2θ ))) represents the input feature map H 2θ After the convolution operation, the Relu activation function operation is performed and the convolution operation is performed at the end; ConvTranspose2d (H 3θ ) represents the input feature map H 3θ Perform deconvolution operation;

[0055] The enhancement module can enhance the input low-resolution and high-semantic features so that its feature information can be better obtained and help the network obtain detailed information of the image.

[0056] Step 2.4 includes the following steps:

[0057] S2.4.1: Build an entry module for processing foggy images as input to the generator;

[0058] Build a convolution operation Conv with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, followed by a layer of ReLU activation function. The foggy image input to the generator G passes through this entry module and outputs a foggy feature with unchanged spatial resolution and 16 channels. Figure X ;

[0059] S2.4.2: Build a downsampling encoder for learning haze features;

[0060] S2.4.3: Add an enhancement module to enhance the low-resolution high semantic feature representation capability after the fifth layer encoder to obtain enhanced features Figure X 5′;

[0061] S2.4.4: Build an upsampling decoder for reconstructing the dehazed image;

[0062] S2.4.5: Build the network layer that outputs the results of the generator.

[0063] S2.4.2 includes the following steps:

[0064] S2.4.2.1: Construct a module that is composed of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2, which are added in parallel to form the first layer of the downsampling encoder. This module outputs the haze image features with unchanged resolution. Figure X 1. Similarly, the module is stacked four times in sequence to form the second, third, fourth, and fifth layers of the downsampling encoder;

[0065] S2.4.2.2: Create a downsampling operation between adjacent downsampling encoder layers, specifically, perform a convolution Conv operation with a kernel size of 3×3, a stride of 2, and a padding of 0 on the feature map output by the encoder layer of the previous layer. Therefore, the second, third, fourth, and fifth layers of the downsampling encoder output image feature maps with half the resolution and double the number of channels, denoted as X2, X3, X4, and X5 respectively;

[0066] S2.4.3: Add an enhancement module to enhance the low-resolution high semantic feature representation capability after the fifth layer encoder to obtain enhanced features Figure X 5′.

[0067] S2.4.4 includes the following steps:

[0068] S2.4.4.1: A module composed of five parallel operations of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2 is stacked in sequence to form the fifth, fourth, third, second, and first layers of the upsampling decoder of the generative network;

[0069] S2.4.4.2: An upsampling operation is constructed between adjacent upsampling decoder levels. Specifically, a transposed convolution operation with a kernel size of 2×2 and a stride of 2 is performed, which doubles the resolution of the input feature map and halves the number of channels.

[0070] S2.4.4.3: Let X5′ be the input of the fifth layer of the upsampling decoder. After being processed by the fifth layer of the upsampling decoder, the feature map with the same resolution as the input is output. After one upsampling operation, the feature map with doubled resolution and half the number of channels is obtained, which is recorded as Y4;

[0071] S2.4.4.4: Let the feature map of Y4 and X4 be the input of the fourth layer of the upsampling decoder. After processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y3;

[0072] S2.4.4.5: Let the feature map of Y3 and X3 be the input of the third layer of the upsampling decoder. After being processed by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y2;

[0073] S2.4.4.6: Let the feature map of Y2 and X2 be the input of the third layer of the upsampling decoder. After processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y1;

[0074] S2.4.4.7: Let the feature map of Y1 and X1 be the input of the first layer of the upsampling decoder. After processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map with unchanged resolution and number of channels is obtained, which is recorded as Y.

[0075] In S2.4.5, build the network layer for outputting the results of the generator:

[0076] Build a convolution block Conv with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, followed by a layer of ReLU activation function. After Y is calculated by Conv and ReLU, the generator completes the defogging operation of the input foggy image and outputs a defogging clear image with three RGB channels.

[0077] Step 2.5 includes the following steps:

[0078] S2.5.1: Build the entry module of the discriminator:

[0079] Build a concat splicing function, a convolution operation Conv with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a LeakyReLU activation function. After the real clear image and the defogging generated image of the input discriminator D pass through this entry module, the output spatial resolution is halved and the number of channels is 64.

[0080] S2.5.2: Build three main modules of the discriminator in series. Each module consists of a convolution operation Conv with a kernel size of 4×4, a stride of 2, and a padding of 1, a batch normalization layer, and a LeakyReLU activation function layer. After the feature map passes through this module, the output spatial resolution is halved and the number of channels is doubled. Finally, the discriminator outputs a discriminant matrix with a spatial resolution of 30×30 and a channel number of 1.

[0081] The mean value of the discriminator output matrix is ​​used as the True / False output. An output value of 1 indicates that the image input to the discriminator is a real fog-free image, and an output value of 0 indicates that the image input to the discriminator is a fog-free image generated by the generator.

[0082] In step 3, the training data set in step 1 is used to train the multi-depth fusion adversarial generative network MDF-GAN built in step 2 to optimize the model parameters.

[0083] During the training process, the multi-depth fusion adversarial generative network MDF-GAN optimizes the loss function and adjusts the weight values ​​in the network to minimize the calculation result of the loss function, thereby achieving the best convergence effect.

[0084] The step 3 comprises the following steps:

[0085] S3.1: Initialize the hyperparameters required for training the model, the batch size, training rounds, and initial learning rate of the training set;

[0086] Start training the model. Input the haze image to be defogged into the generator in MDF-GAN. Each time, the defogged image generated by the generator and its corresponding real haze-free image are input into the discriminator in the adversarial network. The discriminator will output 0 or 1, and repeat this adversarial generation learning process.

[0087] S3.2: Construct a loss function for MDF-GAN to make the detailed information of the image generated by the generator more complete, prevent the generator from being weaker than the discriminator, resulting in an unstable training process and low learning efficiency of the generator. Therefore, the specific expression of the generator loss function using the multi-loss fusion scheme is as follows:

[0088]

[0089] Where, L gen represents the total loss of the generator; L con Indicates content loss; L P Indicates perceived loss; L adv Represents the loss of resistance; K m,i,j represents a real haze-free image with m channels and i×j size; m,i,j Represents a dehazed image with m channels and i×j size; It represents the feature map output by a real fog-free image with m channels and size i×j after passing through n layers of convolutional neural network; represents the feature map of the defogging image with m channels and size i×j after passing through n layers of convolutional neural network; D(x,y) represents the result obtained after the foggy image and the defogging image are input to the discriminator; D(x,K) represents the result obtained after the foggy image and the real fog-free image are input to the discriminator; E x,y [D(x,y)] represents the expected value of the result obtained after the foggy image and the defogged image are input into the discriminator in the joint distribution samples of the foggy image and the defogged image; E x,K [D(x,K)] represents the expected value of the result obtained after the foggy image and the real fog-free image are input into the discriminator in the joint distribution samples of the foggy image and the real fog-free image;

[0090] E[*] represents the expected value of the sample distribution, ||*||1 represents the L1 norm, x represents the foggy image input to the generator, y represents the image after the generator completes the defogging, K represents the real fog-free image for comparison, C, W, H represent the channel, width and height of the feature map respectively, represents the nth layer feature map of the convolutional neural network. The present invention selects the 7th layer feature map of the vgg-16 network. D(*) represents the discriminator. D∈1-Lipschitz is the limiting condition of the discriminator parameters. The present invention constrains the discriminator parameters between (-0, 01, 0, 01).

[0091] The task of the discriminator is to distinguish the real haze-free image K from the haze-free image y generated by the generator, that is, to output 1 for the real haze-free image K and 0 for the haze-free image y generated by the generator;

[0092] Therefore, the loss function of the discriminator uses the cross entropy binary classification loss function, denoted by L D , the specific calculation process is as follows:

[0093]

[0094] In the formula, ω n is the weight, the default value is 1; K n Represents the probability of this type of output, y n represents the true value of the class; l1,…,l N They represent the N loss values ​​calculated after N sets of real haze-free images and corresponding dehazed images input into the discriminator; T represents the transpose of the vector.

[0095] In the step 4, the real foggy image that needs to be dehazed is input into the trained multi-depth fusion adversarial generative network MDF-GAN, and the image is processed by the generator of the multi-depth fusion adversarial generative network MDF-GAN to output a clear image with unchanged resolution, natural color, and no fog.

[0096] The present invention provides an image defogging method based on multi-depth fusion of conditional generative adversarial networks, and the technical effects are as follows

[0097] 1) The present invention combines the extracted underlying detail features with the high-level global features to improve the fineness of the dehazed image. It adopts the adversarial network idea and reduces the difficulty of network training. It is suitable for image dehazing tasks such as image dehazing and can more accurately sense the concentration and size of haze. It effectively solves the problem of uneven haze distribution concentration in real haze situations, which leads to poor global dehazing effect in previous methods. It can retain the original details and colors as much as possible while dehazing.

[0098] 2) The present invention adopts an end-to-end technical design to learn the mapping relationship between foggy images and fog-free images, and uses a large number of paired foggy and fog-free image datasets as data drive.

[0099] 3) The present invention takes a short time to process haze images, has a fast response, and has a good defogging effect on continuous areas.

[0100] 4) The present invention improves the fineness of the dehazed image by fusing the extracted underlying detail features with the high-level global features, adopts the adversarial network idea, reduces the difficulty of network training, and is suitable for image dehazing tasks.

[0101] 5) For the entire generator network, feature extraction is the key to complete haze removal. Given that there are different concentrations of fog in haze images, some information in the image is lost and difficult to restore. The haze learning unit constructed in step 2.1 of the present invention can improve the haze removal ability of the image, so that the generator can strengthen the learning of haze features to achieve better removal of haze in the image.

[0102] 6) In the encoder and decoder of the generator, the feature maps of different levels carry different information and have different dehazing effects, but the lack of information interaction between non-adjacent levels reduces the dehazing ability. The effect of simply using the splicing method to fuse the features of non-adjacent levels is often poor. The multi-depth projection module constructed in step 2.2 of the present invention can solve the problem of spatial information loss during upsampling and downsampling and the problem of information loss between non-adjacent levels, thereby improving the image restoration ability of the network.

[0103] 7) The enhancement module constructed in step 2.3 of the present invention is located between the encoder and decoder of the generator. Due to the multiple downsampling of the encoder, the high semantic features in the image are more prominent. These features are crucial for the image defogging task. The enhancement module constructed in step 2.3 of the present invention can not only enhance the feature expression ability of the network, so that the network can learn these high semantic features, but also help the model capture more generalized features. In general, it is not only responsible for extracting and strengthening the high semantic features transmitted by the encoder, but also responsible for transmitting these features to the decoder.

[0104] 8) The generator constructed in 2.4 of the present invention is the key to restore the foggy image to the corresponding clear image, and its performance will be directly related to the effect of image defogging. The generator can learn the complex mapping from foggy images to clear images and generate high-quality defogging results. Moreover, through the end-to-end learning process, it can not only retain image details and restore natural colors, but also adapt to different fog conditions and scenes, showing high flexibility and robustness.

[0105] 9) The discriminator constructed in step 2.5 of the present invention guides the training direction of the generator by providing competitive feedback signals, thereby ensuring the quality and diversity of the generated data. Moreover, the discriminator can evaluate the difference between the generated image and the real image, forcing the generator to produce more realistic samples, while improving the model's sensitivity to details and overall structure, and enhancing the credibility of the generated image. In addition, the existence of the discriminator also helps to stabilize the training process, prevent the generator from converging to a suboptimal solution too early, and ensure that the generative adversarial network can generate high-quality, high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 It is a flow chart of the present invention.

[0107] Figure 2 This is a diagram of the multi-depth fusion adversarial generation network structure of the present invention.

[0108] Figure 3 This is a structural diagram of the haze learning unit of the present invention.

[0109] Figure 4 This is a structural diagram of the multi-depth projection module of the present invention.

[0110] Figure 5 This is a structural diagram of the enhancement module of the present invention.

[0111] Figure 6 This is a structural diagram of the generator of the present invention.

[0112] Figure 7 This is a structural diagram of the discriminator of the present invention.

[0113] Figure 8 This is a comparison chart of the visualization results of the present invention and other dehazing methods on the RESIDE outdoor dataset.

[0114] Fig. 9 This is a comparison chart of the visualization results of the present invention and other dehazing methods on the RESIDE indoor dataset.

[0115] Fig.10 This is a comparison chart of the visualization results of the present invention and other dehazing methods on the RESIDE real scene dataset. DETAILED DESCRIPTION

[0116] like Figure 1 to Figure 7 As shown, an embodiment of the present invention provides an image defogging method based on a conditional generative adversarial network with multiple depth fusion, comprising the following steps:

[0117] S1: Obtain a data set including foggy images and clear images corresponding to the foggy images one by one, and divide the data set into a training set and a test set;

[0118] S2: A multi-depth fusion image dehazing model is constructed based on the framework of conditional generative adversarial network. The model includes a generator and a discriminator. The generator is responsible for generating dehazed images, and the discriminator is responsible for identifying whether the input image is a real image or a generated image and providing feedback to the generator based on the output results.

[0119] S3: Use the training set to train the multi-depth fusion image dehazing model constructed in S2 to obtain a trained image dehazing model.

[0120] S4: Use the trained image dehazing model to test the test set to obtain the dehazed image.

[0121] In this embodiment, specifically, in S1, a foggy image dataset for training is constructed. First, 13,990 indoor foggy images generated from 1,399 clear images and 8,970 outdoor foggy images generated from 897 clear images are selected from the open source real single image defogging dataset RESIDE as the original data source. Then, the dataset is divided into a training set and a test set in a ratio of 8:2.

[0122] In this embodiment, specifically, in S2, a multi-deep fusion adversarial generation network MDF-GAN (Multi-deep fusion adversarial generation network) for image dehazing is constructed;

[0123] Build a multi-depth fusion generator for generating near-real images. The generator can learn the mapping from foggy images to their corresponding real fog-free images. The generator has an upsampling encoder composed of a Haze Learning Block (HLB) that can extract features of foggy images, a Multi-depth Projection Module (MDPM) that fuses semantic features of different depths, a downsampling decoder, and an enhancement module that can enhance the representation capability of low-resolution high-semantic features.

[0124] Build a discriminator to help the generator achieve the best fit. Provide the generator's output and its real haze-free image as input to the discriminator for comparison. The discriminator then feeds back the quality judgment score to the generator.

[0125] The specific steps are as follows:

[0126] S2.1: Construct a haze learning unit for extracting features of hazy images;

[0127] S2.2: Construct a multi-depth projection module for fusing different depth semantic features;

[0128] S2.3: Build an enhancement module to enhance the low-resolution high-semantic feature representation capability;

[0129] S2.4: Build a multi-depth fusion generator for generating images close to real images;

[0130] S2.5: Build a discriminator to help the generator achieve the best fit.

[0131] In this embodiment, specifically, in S3, it is necessary to use the training data set in S1 to train the multi-depth fusion adversarial generative network MDF-GAN built in S2, and optimize the model parameters through the back propagation algorithm;

[0132] During the training process, MDF-GAN optimizes the loss function and adjusts the weight values ​​in the network to minimize the calculated result of the loss function, thereby achieving the best convergence effect.

[0133] In this embodiment, specifically, in S4, a real foggy image that needs to be defogged is input into the trained MDF-GAN, and the image is processed by the generator of the MDF-GAN to output a clear image with unchanged resolution, natural color, and no fog.

[0134] In this embodiment, specifically, in S2.1, the specific steps of constructing a haze learning unit HLB for extracting features of a hazy image are as follows:

[0135] S2.1.1: Build a Globally Aware Module (GAM). First, perform Conv convolution with a kernel size of 2×2, a step size of 1, and a padding of 1, a depthwise separable convolution with a kernel size of 7×7, a step size of 3, and a group number of 3, and batch normalization on the feature map Input, and output a feature map F1∈R with unchanged resolution and number of channels. C×H×W , where C, H, and W represent the number of channels, height, and width, respectively. Then the feature map is parallelly input into the AdaptiveAvgPool2d global average pooling and AdaptiveMaxPool2d global maximum pooling with a length and width of 1×1, and the results are concatenated to obtain the feature map T m , and then the feature map T m The Conv convolution and Sigmoid activation function with a convolution kernel size of 2×2, a step size of 1, and a padding of 1 are input in parallel to obtain the weight value ω. Finally, ω is multiplied by the feature map F1 to obtain the feature map ωF1 with half the resolution and number of channels.

[0136] The output process of the output feature map ωF1 is expressed as:

[0137]

[0138] S2.1.2: Build a dense module DenseNet consisting of four DenseBlocks. Each DenseBlock consists of a Conv convolution with a convolution kernel size of 2×2, a step size of 1, and a padding of 1, and a PReLU activation function. Each DenseBlock is densely connected. First, the feature map ωF1 is input into DenseNet, and then a Conv convolution operation with a convolution kernel size of 1×1 is performed. Finally, it is residually connected with the feature map ωF1 to output the feature map F2 with unchanged resolution and number of channels.

[0139] The feature map F2 can convey to the network whether a certain area in the image is important and the degree to which it needs to be defogged, allowing the network to focus on the dense fog area. Therefore, the output process of the haze learning unit HLB is expressed as:

[0140]

[0141] The haze learning unit HLB is used for defogging by region and degree. This unit can focus on the details in the haze image, determine the fog concentration in the captured haze image, locate the area in the image that needs to be converted, effectively defogging while avoiding over-defogging.

[0142] In this embodiment, specifically, an embodiment of S2.1 is given below:

[0143] Input the feature map Input with a spatial resolution of 128×128 and a number of channels of 32 into the global perception module GAM to obtain a feature map ωF1 with a resolution of 128×128 and a number of channels of 16;

[0144] The feature map ωF1 with a spatial resolution of 128×128 and 16 channels is input into the dense module DenseNet composed of four DenseBlocks. After the first DenseBlock, a feature map F2 with a spatial resolution of 128×128 and 16 channels is obtained.

[0145] The feature map ωF1 and the feature map F2 are concatenated in the channel dimension to obtain a feature map with a spatial resolution of 128×128 and a number of channels of 32, which is input into the second DenseBlock to obtain a feature map F3 with a spatial resolution of 128×128 and a number of channels of 16;

[0146] The feature maps ωF1, F2, and F3 are concatenated in the channel dimension to obtain 48 feature maps with a spatial resolution of 128×128 and a number of channels, which are input into the third DenseBlock to obtain a feature map F4 with a spatial resolution of 128×128 and a number of channels of 16;

[0147] The feature maps ωF1, F2, F3, and F4 are concatenated in the channel dimension to obtain a feature map with a spatial resolution of 128×128 and 64 channels, which is input into the fourth DenseBlock to obtain a feature map F5 with a spatial resolution of 128×128 and 16 channels;

[0148] The feature maps ωF1, F2, F3, F4, and F5 are concatenated in the channel dimension to obtain a feature map with a spatial resolution of 128×128 and 80 channels, which is input into the Conv convolution with a convolution kernel size of 1×1 to obtain a feature map F6 with a spatial resolution of 128×128 and 16 channels. Then, the feature map F6 and the feature map ωF1 are concatenated in the channel dimension to obtain a feature map F with a spatial resolution of 128×128 and 32 channels. output . F output This is the Output of the HLB learning unit on haze.

[0149] In this embodiment, specifically, the convolution operation is the most important and most common operation part of the convolutional neural network, and parameters such as step size, convolution kernel size, and padding value need to be set for it. The earliest convolutional neural network model LeNet uses 3 convolution layers. The activation function can enhance the nonlinear expression of the neural network and learn more complex information. The ReLU activation function is a nonlinear function, and its derivative is simpler, allowing the model to converge faster. Batch Normalization is also a commonly used technology in neural networks. Its main function is to accelerate the convergence speed of the neural network, improve the stability of the model, and help prevent overfitting;

[0150] The feature map is the result of one or more convolutions, pooling and other operations on the input image in the convolutional neural network. One layer of the feature map corresponds to the result of one convolution of the input image. Therefore, the feature map after convolution is equal to the number of convolution kernels, that is, the number of channels of the output feature map after convolution operation is determined by the number of convolution kernels set during the convolution operation.

[0151] Residual connection is a commonly used technique in deep neural networks. By adding the output of the previous layer directly to the input of the current layer, the network can learn the residual (i.e., the change of the current layer relative to the previous layer), making it easier to train deep neural networks.

[0152] In this embodiment, specifically, in S2.2, the specific steps of constructing a multi-depth projection module MDPM for fusing different depth semantic features are as follows:

[0153] S2.2.1: Build an Efficient Channel Attention (ECA) mechanism to fuse important features between non-adjacent layers. First, perform AdaptiveAvgPool2d global average pooling on the output feature map F2 of the haze learning unit HLB, then perform a one-dimensional convolution Conv1d operation with adaptive input size and nonlinear enhancement of the Sigmoid activation function to obtain the channel dimension weight α. Finally, multiply the channel dimension weight α and the feature map F2 to obtain the weight screening map αF2 with unchanged resolution and number of channels.

[0154] The output process of the weight screening graph αF2 is expressed as:

[0155]

[0156] S2.2.2: Build a multi-depth projection module MDPM for fusing semantic features of different depths. First, perform a Conv convolution operation with a kernel size of 4×4, a step size of 2, and a padding of 1 on the input feature map Input, and a nonlinear enhancement of the ReLU activation function. Then, subtract it from the weight screening map αF2, and perform a ConvTranspose2d deconvolution operation with a kernel size of 4×4, a step size of 2, and a padding of 1, and a nonlinear enhancement of the ReLU activation function. Then, add it to the input feature map Input to obtain the output feature map J1 with unchanged resolution and number of channels.

[0157] The output process of the output feature map J1 is expressed as:

[0158]

[0159] Feature maps at different levels carry different information and have different dehazing effects. However, the lack of information interaction between non-adjacent levels reduces the dehazing ability. Simply using the splicing method to fuse features of non-adjacent levels often has poor results. The multi-depth projection module MDPM can fuse semantic features of different depths and solve the above problem.

[0160] In this embodiment, specifically, an embodiment of S2.2 is given below:

[0161] Input the feature map Input with a spatial resolution of 128×128 and a number of channels of 32 into the channel attention mechanism ECA to obtain a feature map αF1 with a resolution of 128×128 and a number of channels of 32;

[0162] The feature map αF1 with a spatial resolution of 128×128 and 32 channels is input into the Conv convolution with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and the nonlinear enhancement of the ReLU activation function. Secondly, it is subtracted from the weight screening map αF2 to obtain a feature map with a spatial resolution of 256×256 and 32 channels. Then it is input into the ConvTranspose2d deconvolution with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and the nonlinear enhancement of the ReLU activation function to obtain a feature map with a spatial resolution of 128×128 and 32 channels. Finally, it is added to the feature map αF1 to obtain the feature map J1 with a spatial resolution of 128×128 and 32 channels.

[0163] In this embodiment, specifically, in S2.3, the specific steps of constructing an enhancement module for enhancing the low-resolution high semantic feature representation capability are as follows:

[0164] S2.3.1: Build a residual group consisting of 3 residual blocks connected in series. Each residual block consists of a Conv convolution with a kernel size of 3×3 and a stride of 1, a ReLU activation function, a Conv convolution with a kernel size of 3×3 and a stride of 1, and a residual connection.

[0165] S2.3.2: Build the enhancement module. First, perform a ConvTranspose2d deconvolution operation on the input with a convolution kernel size of 4×4, a step size of 2, and a padding of 1. Then perform 6 residual group operations in sequence to obtain an enhanced feature map H with unchanged resolution and number of channels.

[0166] The enhancement process of the enhanced feature map H is expressed as:

[0167]

[0168] The enhancement module can enhance the input low-resolution and high-semantic features so that its feature information can be better obtained and help the network obtain detailed information of the image.

[0169] In this embodiment, specifically, an embodiment of S2.3 is given below:

[0170] The feature map Input with a spatial resolution of 16×16 and a number of channels of 256 is input into the ConvTranspose2d deconvolution with a convolution kernel size of 4×4, a stride of 2, and a padding of 1, and then input into the first of the six residual groups in the enhancement module to obtain a feature map H1 with a spatial resolution of 16×16 and a number of channels of 256;

[0171] Input the feature map H1 with a spatial resolution of 16×16 and a number of channels of 256 into the second residual group to obtain a feature map H2 with a spatial resolution of 16×16 and a number of channels of 256;

[0172] Input the feature map H2 with a spatial resolution of 16×16 and a number of channels of 256 into the third residual group to obtain a feature map H3 with a spatial resolution of 16×16 and a number of channels of 256;

[0173] Input the feature map H3 with a spatial resolution of 16×16 and a number of channels of 256 into the fourth residual group to obtain a feature map H4 with a spatial resolution of 16×16 and a number of channels of 256;

[0174] Input the feature map H4 with a spatial resolution of 16×16 and a number of channels of 256 into the fifth residual group to obtain a feature map H5 with a spatial resolution of 16×16 and a number of channels of 256;

[0175] Input the feature map H5 with a spatial resolution of 16×16 and a number of channels of 256 into the sixth residual group to obtain a feature map H6 with a spatial resolution of 16×16 and a number of channels of 256;

[0176] In this embodiment, specifically, in S2.4, the specific steps of constructing a generator for generating multi-depth fusion close to real images are as follows:

[0177] S2.4.1: Build the entry module for processing foggy images as input to the generator:

[0178] Build a convolution operation Conv with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, followed by a layer of ReLU activation function. The foggy image input to the generator G passes through this entry module and outputs a foggy feature with unchanged spatial resolution and 16 channels. Figure X ;

[0179] S2.4.2: Build a downsampling encoder for learning haze features:

[0180] S2.4.2.1: Construct a module that is composed of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2, which are added in parallel to form the first layer of the downsampling encoder. This module outputs the haze image features with unchanged resolution. Figure X 1. Similarly, the module is stacked four times in sequence to form the second, third, fourth, and fifth layers of the downsampling encoder;

[0181] S2.4.2.2: Create a downsampling operation between adjacent downsampling encoder layers, specifically, perform a convolution Conv operation with a kernel size of 3×3, a stride of 2, and a padding of 0 on the feature map output by the encoder layer of the previous layer. Therefore, the second, third, fourth, and fifth layers of the downsampling encoder output image feature maps with half the resolution and doubled the number of channels, denoted as X2, X3, X4, and X5 respectively;

[0182] S2.4.3: Add an enhancement module to enhance the low-resolution high semantic feature representation capability after the fifth layer encoder to obtain enhanced features Figure X 5′;

[0183] S2.4.4: Build an upsampling decoder for reconstructing the dehazed image:

[0184] S2.4.4.1: A module composed of five parallel operations of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2 is stacked in sequence to form the fifth, fourth, third, second, and first layers of the upsampling decoder of the generative network;

[0185] S2.4.4.2: An upsampling operation is constructed between adjacent upsampling decoder levels. Specifically, a transposed convolution operation with a kernel size of 2×2 and a stride of 2 is performed, which doubles the resolution of the input feature map and halves the number of channels.

[0186] S2.4.4.3: Let X5′ be the input of the fifth layer of the upsampling decoder. After being processed by the fifth layer of the upsampling decoder, the feature map with the same resolution as the input is output. After one upsampling operation, the feature map with doubled resolution and half the number of channels is obtained, which is recorded as Y4;

[0187] S2.4.4.4: Let the feature map of Y4 and X4 be the input of the fourth layer of the upsampling decoder. After processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y3;

[0188] S2.4.4.5: Let the feature map of Y3 and X3 be the input of the third layer of the upsampling decoder. After being processed by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y2;

[0189] S2.4.4.6: Let the feature map of Y2 and X2 be the input of the third layer of the upsampling decoder. After processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y1;

[0190] S2.4.4.7: Let the feature map of Y1 and X1 be the input of the first layer of the upsampling decoder. After being processed by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map with unchanged resolution and number of channels is obtained, which is denoted as Y;

[0191] S2.4.5: Build the network layer for outputting the results of the generator:

[0192] Build a convolution block Conv with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, followed by a layer of ReLU activation function. After Y is calculated by Conv and ReLU, the generator completes the defogging operation of the input foggy image and outputs a defogging clear image with three RGB channels.

[0193] In this embodiment, specifically, an embodiment of S2.4 is given below:

[0194] Input the foggy image with 256×256 spatial resolution and 3 channels into the generator’s entry module to obtain features with 256×256 spatial resolution and 16 channels. Figure X ;

[0195] The feature with 256×256 spatial resolution and 16 channels Figure X Input to the first layer of the encoder, and get the features of 256×256 spatial resolution and 16 channels Figure X 1;

[0196] The feature with 256×256 spatial resolution and 16 channels Figure X 1 is input to the second layer of the encoder, and then downsampled to obtain a feature with a spatial resolution of 128×128 and a channel number of 32. Figure X 2;

[0197] The feature with 128×128 spatial resolution and 32 channels Figure X 2 is input to the third layer of the encoder, and then downsampled to obtain a feature with a spatial resolution of 64×64 and a number of channels of 64 Figure X 3;

[0198] The feature with 64×64 spatial resolution and 64 channels Figure X3 is input to the fourth layer of the encoder, and then downsampled to obtain a feature with a spatial resolution of 32×32 and a channel number of 128 Figure X 4;

[0199] The feature with 32×32 spatial resolution and 128 channels Figure X 4 is input to the fifth layer of the encoder, and then downsampled to obtain a feature with a spatial resolution of 16×16 and a channel number of 256 Figure X 5;

[0200] The feature with 16×16 spatial resolution and 256 channels Figure X 5 is input into the enhancement module to obtain features with 16×16 spatial resolution and 256 channels Figure X 5′;

[0201] The feature with 16×16 spatial resolution and 256 channels Figure X 5′ is input into the fifth layer of the decoder, and then upsampled to obtain a feature map Y4 with a spatial resolution of 32×32 and a channel number of 128;

[0202] The feature map Y4 with a spatial resolution of 32×32 and a number of channels of 128 and the feature Figure X 4 is added and fused bit by bit and then input into the fourth layer of the decoder. After upsampling, a feature map Y3 with a spatial resolution of 64×64 and 64 channels is obtained.

[0203] The feature map Y3 with a spatial resolution of 64×64 and 64 channels and the feature Figure X 3 After bitwise addition and fusion, the data is input into the third layer of the decoder, and then upsampled to obtain a feature map Y2 with a spatial resolution of 128×128 and a channel number of 32;

[0204] The feature map Y2 with a spatial resolution of 128×128 and a number of channels of 32 and the feature Figure X 2 After bitwise addition and fusion, it is input into the second layer of the decoder, and then upsampled to obtain the feature map Y1 with a spatial resolution of 256×256 and 16 channels;

[0205] The feature map Y1 with a spatial resolution of 256×256 and 16 channels and the feature Figure X 1 After bitwise addition and fusion, it is input into the first layer of the decoder, and then upsampled to obtain a feature map Y with a spatial resolution of 256×256 and 16 channels;

[0206] The feature map Y with a spatial resolution of 256×256 and 16 channels is input into the output layer of the generator to obtain a dehazed image with a spatial resolution of 256×256 and 3 channels.

[0207] In this embodiment, specifically, in S2.5, the specific steps of building a discriminator for helping the generator to achieve the best fitting state are as follows:

[0208] S2.5.1: Build the entry module of the discriminator:

[0209] Build a concat splicing function, a convolution operation Conv with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a LeakyReLU activation function. After the real clear image and the defogging generated image of the input discriminator D pass through this entry module, the output spatial resolution is halved and the number of channels is 64.

[0210] S2.5.2: Build three main modules of the discriminator in series. Each module consists of a convolution operation Conv with a kernel size of 4×4, a stride of 2, and a padding of 1, a batch normalization layer, and a LeakyReLU activation function layer. After the feature map passes through this module, the output spatial resolution is halved and the number of channels is doubled. Finally, the discriminator outputs a discriminant matrix with a spatial resolution of 30×30 and a channel number of 1.

[0211] The mean value of the discriminator output matrix is ​​used as the True / False output. An output value of 1 indicates that the image input to the discriminator is a real fog-free image, and an output value of 0 indicates that the image input to the discriminator is a fog-free image generated by the generator.

[0212] In some embodiments, in S3, the multi-depth fusion image defogging model constructed in S2 is trained using the training set to obtain the trained image defogging model in the following specific steps:

[0213] S3.1: Initialize the hyperparameters required for training the model, the batch size, training rounds, and initial learning rate of the training set;

[0214] Start training the model. Input the haze image to be defogged into the generator in MDF-GAN. Each time, the defogged image generated by the generator and its corresponding real haze-free image are input into the discriminator in the adversarial network. The discriminator will output 0 or 1, and repeat this adversarial generation learning process.

[0215] S3.2: Construct a loss function for MDF-GAN to make the detailed information of the image generated by the generator more complete, prevent the generator from being weaker than the discriminator, resulting in an unstable training process and low learning efficiency of the generator. Therefore, the specific expression of the generator loss function using the multi-loss fusion scheme is as follows:

[0216]

[0217] Where E[*] represents the expected value of the sample distribution, ‖*‖1 represents the L1 norm, x represents the foggy image input to the generator, y represents the image after the generator completes the defogging, K represents the real fog-free image for comparison, C, W, H represent the channel, width and height of the feature map respectively, represents the nth layer feature map of the convolutional neural network. In this paper, it is selected as the 7th layer feature map of the vgg-16 network. D(*) represents the discriminator. D∈1-Lipschitz is the limiting condition of the discriminator parameters. The present invention constrains the discriminator parameters between (-0, 01, 0, 01).

[0218] The task of the discriminator is to distinguish the real haze-free image K from the haze-free image y generated by the generator, that is, to output 1 for the real haze-free image K and 0 for the haze-free image y generated by the generator;

[0219] Therefore, the loss function of the discriminator uses the cross entropy binary classification loss function, denoted by L D , the specific calculation process is as follows:

[0220]

[0221] In the formula, y represents the image after the generator completes the defogging, K represents the real fog-free image for comparison, ω n is the weight, the default value is 1, K n Represents the probability of this type of output, y n Represents the truth value of this class.

[0222] The back propagation algorithm is an existing optimization algorithm for training neural networks. It can continuously adjust the weights in the neural network to minimize the error between the model's predicted output and the actual label.

[0223] Verification example:

[0224] In order to verify the performance of the image dehazing method based on conditional generative adversarial network and multi-depth fusion proposed by the present invention, the present invention uses the artificially synthesized haze image data with reference and real haze image data of various types without reference in the public image dehazing dataset RESIDE (REalistic Single Image Dehazing) to conduct haze image clarity experiments. In order to comprehensively and objectively evaluate the performance of the proposed image dehazing method, the image dehazing effects of the proposed method and several classic and recently developed algorithms in the field of image dehazing are compared; these algorithms are: DCP, CAP, Dehazed, AOD-Ne, GridDehazed, FFA-Net, FD-GAN, HIDE-GAN, SADNet, GANID, and the experimental results are compared and analyzed using objective evaluation numerical indicators.

[0225] like Figure 8 and Fig. 9 As shown in FIG. 1 , the comparison of the clarity effect of the method of the present invention and other comparison algorithms on artificially synthesized haze images with reference in outdoor and indoor scenes in the RESIDE dataset is shown:

[0226] Compared with the real fog-free image, the image processed by DCP is darker overall. At the same time, some areas that do not meet the dark channel prior information will have serious distortion. For example, the color of the sky area in the image is obviously inconsistent with the original image.

[0227] The CAP algorithm removes fog by comparing the brightness and saturation differences between foggy and non-fog areas. The processed images have poor effects and all images are distorted.

[0228] The HIDE-GAN algorithm can achieve good defogging, but the defogging effect is not ideal for images with thick fog, and the overall color restoration of the defogged image is not high;

[0229] The GridDehazeNet algorithm has significantly improved image contrast and saturation due to the use of an attention-based multi-scale network, but it still has the problem of incomplete dehazing, and the image restoration degree needs to be improved in terms of details;

[0230] The FFA-Net algorithm achieved satisfactory results in terms of both overall effect and color restoration, but failed to effectively handle image details, such as haze in the deep areas of the scene;

[0231] The SADNet algorithm is a defogging method based on the attention mechanism, but a lot of haze can still be seen in the processed pictures, and the effect is not ideal. The pictures processed by the method of the present invention not only successfully defog the pictures, but also improve the color saturation, make the colors more vivid, the details richer, and the brightness differences in different areas more obvious, making the images more layered.

[0232] In order to more objectively evaluate the clearing effects of different algorithms on the synthesized haze images, the reference evaluation indicators peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to statistically analyze the images after being cleared by several comparison algorithms. The statistical results are shown in Table 1. The reference evaluation indicators peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) can reflect the distortion of the haze images after clearing compared with the clear images not polluted by haze. The larger the values ​​of the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), the better the performance of the corresponding haze image clearing algorithm.

[0233] The PSNR calculation formula is as follows:

[0234]

[0235] Where W and H represent the width and height of the image, K(i,j) represents the real haze-free image of size i×j, I(i,j) represents the generated dehazed image of size i×j, MSE represents the mean square error between all real haze-free images and generated dehazed images, ||K(i,j)-I(i,j)‖ 2 represents the square of the difference between the real haze-free image of size i×j and the generated dehazed image of size i×j, PSNR represents the peak signal-to-noise ratio, and lg(i) represents the logarithm with base 10.

[0236] The SSIM calculation formula is as follows:

[0237]

[0238] Where y and K are the dehazed image and the real haze-free image, respectively, μ y and μ K is the variance of y and K, σ yK is the covariance of y and K, c1 and c2 are two constants to prevent the denominator from being 0 during calculation, and SSIM represents structural similarity.

[0239] Table 1 Comparison of objective evaluation indicators of different algorithms for the clarity of artificially synthesized haze images

[0240]

[0241] In the comparative experiments, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) performances of several comparative algorithms are basically consistent with the visual subjective evaluation experimental results. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the DCP method perform worst in indoor scenes, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the CAP method perform worst in outdoor scenes. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the method of the present invention are the best for both outdoor and indoor scene test images.

[0242] In order to further verify the performance of the proposed method in the present invention for the clarity of real environment haze images, Fig.10 As shown in FIG. 1 , the comparison experiment of the clarity effect of the method of the present invention and other comparison algorithms in the real scene haze image in the RESIDE dataset is shown; the DCP method and the CAP method still cannot restore the image color well, such as Fig.10 The sky part in the second row of the figure; although the HIDE-GAN method can remove a certain amount of haze, the overall brightness of the dehazed image is dark; the dehazing effects of the GridDehazeNet method and the FFA-Net method are satisfactory, but there is a problem of incomplete restoration of image details, such as Fig.10 The white cloud part in the third row; the defogging effect of the SADNet method is not successful, and residual fog can still be observed; the overall defogging effect of the method of the present invention is better, and the performance in details and color is also satisfactory.

[0243] Based on the clarity effect of the algorithm of the present invention in processing images with different degrees and types of haze pollution, the algorithm of the present invention has certain advancement and superiority.

Claims

1. An image dehazing method based on multi-depth fusion of conditional generative adversarial networks, characterized by The following steps are involved: Step 1: Obtain a data set containing foggy images and clear images corresponding to the foggy images, and divide the data set into a training set and a test set; Step 2: Based on the framework of conditional generative adversarial network, a multi-depth fusion image defogging model is constructed. The multi-depth fusion image defogging model includes a generator and a discriminator. The generator is used to generate a foggy image. The generator is used to analyze whether the input image is a real image or a generated image, and provide feedback to the generator according to the output result; Step 3: Use the training set to train the multi-depth fusion image dehazing model constructed in step 2 to obtain a trained image dehazing model; Step 4: Use the image dehazing model trained in step 3 to test the test set and obtain the dehazed image.

2. According to claim 1, the image defogging method based on multi-depth fusion of conditional generative adversarial networks is characterized by: In the step 2, a multi-deep fusion adversarial generation network (MDF-GAN) for image defogging is constructed, which specifically includes: Build a generator for generating multi-depth fusion close to real images. The generator can learn the mapping from foggy images to their corresponding real fog-free images. The generator includes a haze learning unit that can extract the features of foggy images. Learning Block (HLB), a multi-depth projection module (MDPM) that fuses semantic features of different depths, an upsampling encoder, a downsampling decoder, and an enhancement module that can enhance the representation capability of low-resolution and high-semantic features; Build a discriminator to help the generator achieve the best fit. Provide the generator's output and its real haze-free image as input to the discriminator for comparison. The discriminator then feeds back the quality judgment score to the generator. The steps include: Step 2.1: Construct a haze learning unit for extracting features of foggy images; Step 2.2: Construct a multi-depth projection module for fusing different depth semantic features; Step 2.3: Build an enhancement module to enhance the low-resolution high-semantic feature representation capability; Step 2.4: Build a multi-depth fusion generator for generating images close to the real image; Step 2.5: Build a discriminator to help the generator achieve the best fitting state.

3. According to claim 2, the image defogging method based on multi-depth fusion of conditional generative adversarial networks is characterized by: Step 2.1 includes the following steps: Step 2.1.1: Build a global perception module GAM: First, the feature map Input is processed with a Conv convolution with a convolution kernel size of 2×2, a stride of 1, and a padding of 1, a depthwise separable convolution with a convolution kernel size of 7×7, a stride of 3, and a group number of 3, and a batch normalization, and the output feature map F1∈R with unchanged resolution and number of channels is C×H×W , where C, H, and W represent the number of channels, height, and width, respectively; then, the feature map is input into the AdaptiveAvgPool2d global average pooling and AdaptiveMaxPool2d global maximum pooling with a length and width of 1×1 in parallel, and the results are concatenated to obtain the feature map T m ; Then, the feature map T m The weight value ω is obtained by inputting the convolution kernel size of 2×2, the step size of 1, and the padding of 1 into the Conv convolution and Sigmoid activation function in parallel; finally, the weight value ω is multiplied by the feature map F1 to obtain the feature map ωF1 with half the resolution and number of channels; The output process of the output feature map ωF1 is expressed as: In the above formula, Conv(Input) means performing a convolution operation on the input feature map Input; Conv(Conv(Input)) means performing a convolution operation on the input feature map Input and then performing a convolution operation again; DsConv(Conv(Input)) means performing a convolution operation on the input feature map Input and then performing a depthwise separable convolution operation; BN(DsConv(Conv(Input))) means performing a convolution operation on the input feature map Input and then performing a depthwise separable convolution operation and finally performing a BatchNormalization operation; MaxPool(F1) means performing a global maximum pooling operation on the feature map F1; AvgPool(F1) means performing a global average pooling operation on the feature map F1; Conv(T m ) represents the feature map T m Perform convolution operation; Sigmoid(Conv(T m )) represents the feature map T m After the convolution operation, the Sigmoid activation function operation is performed; Step 2.1.2: Build a dense module DenseNet composed of four DenseBlocks. Each DenseBlock consists of a Conv convolution with a convolution kernel size of 2×2, a step size of 1, and a padding of 1, and a PReLU activation function. Each DenseBlock is densely connected. First, input the feature map ωF1 into the dense module DenseNet, then perform a Conv convolution operation with a convolution kernel size of 1×1, and finally perform a residual connection with the feature map ωF1 to output a feature map F2 with unchanged resolution and number of channels. The feature map F2 can convey to the network whether a certain area in the image is important and the degree of defogging, so that the network can focus on the dense fog area; therefore, the output process of the Haze Learning Block (HLB) is expressed as: In the above formula, H1 indicates that the feature map ωF1 is obtained after the first DenseBlock operation; H2 indicates that the feature map H1 is obtained after the second DenseBlock operation; H3 indicates that the feature map H2 is obtained after the third DenseBlock operation; H4 indicates that the feature map H3 is obtained after the fourth DenseBlock operation; Relu(Conv(ωF1)) indicates that the feature map ωF1 is convolved and then the Relu activation function is operated; PRelu(Conv(ωF1))) indicates that the feature map ωF1 is convolved and then the PRelu activation function is operated; Conv(H1+ωF1) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed; Relu(Conv(H1+ωF1)) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed, and finally the Relu activation function operation is performed; PRelu(Conv(H1+ωF1))) means that the residual operation is performed on the feature map ωF1 and the feature map H1, and then the convolution operation is performed, and finally the PRelu activation function operation is performed; Conv(H2+H1+ωF1) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed; Relu(Conv(H2+H1+ωF1)) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed, and finally the Relu activation function operation is performed; PRelu(Conv(H2+H1+ωF1))) means that the residual operation is performed on the feature map ωF1, the feature map H1 and the feature map H2, and then the convolution operation is performed, and finally the PRelu activation function operation is performed; Conv(H3+H2+H1+ωF1) means that the residual operation is performed on the feature map ωF1, the feature map H1, the feature map H2 and the feature map H3, and then a convolution operation is performed; Relu(Conv(H3+H2+H1+ωF1)) means that the residual operation is performed on the feature map ωF1, the feature map H1, the feature map H2 and the feature map H3, and then a convolution operation is performed, and finally a Relu activation function operation is performed; PRelu(Conv(H3+H2+H1+ωF1))) means that the residual operation is performed on the feature map ωF1, the feature map H1, the feature map H2 and the feature map H3, and then a convolution operation is performed, and finally a PRelu activation function operation is performed.

4. According to claim 2, the method for image dehazing based on multi-depth fusion of conditional generative adversarial networks is characterized by: Step 2.2 includes the following steps: S2.2.1: Build an Efficient Channel Attention (ECA) mechanism to integrate important features between non-adjacent layers: First, the output feature map F2 of the haze learning unit HLB is globally averaged pooled using AdaptiveAvgPool2d. Then, a one-dimensional convolution Conv1d operation with adaptive input size and nonlinear enhancement of the Sigmoid activation function are performed to obtain the channel dimension weight α. Finally, the channel dimension weight α is multiplied by the feature map F2 to obtain the weighted screening map αF2 with unchanged resolution and number of channels. The output process of the weighted screening map αF2 is expressed as follows: In the above formula, AvgPool(F2) means that the feature map F2 is subjected to global average pooling operation; Conv1d(AvgPool(F2)) means that the feature map F2 is subjected to global average pooling operation and then subjected to one-dimensional convolution operation; Sigmoid(Conv1d(AvgPool(F2))) means that the feature map F2 is subjected to global average pooling operation and then subjected to one-dimensional convolution operation and finally subjected to Sigmoid activation function operation; S2.2.2: Build a multi-depth projection module MDPM for fusing different depth semantic features: First, the input feature map Input is subjected to a Conv convolution operation with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a nonlinear enhancement of the ReLU activation function; secondly, it is subtracted from the weight screening map αF2, and a ConvTranspose2d deconvolution operation with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a nonlinear enhancement of the ReLU activation function is performed together; then, the input feature map Input is added to obtain an output feature map J1 with unchanged resolution and number of channels; the output process of the output feature map J1 is expressed as: In the above formula, T1 represents the feature map obtained by performing convolution operation and Relu activation function operation on the input feature map Input and then subtracting it from the weight screening map αF2; Relu(Conv(Input)) represents performing convolution operation on the input feature map and then performing Relu activation function operation; T2 represents the feature map obtained by performing deconvolution operation and Relu activation function operation on the input feature map T1; ConvTranspose2d(T1) represents performing deconvolution operation on the input feature map T1; Relu(ConvTranspose2d(T1)) represents performing deconvolution operation on the input feature map T1 and then performing Relu activation function operation.

5. According to claim 2, the method for image dehazing based on multi-depth fusion of conditional generative adversarial networks is characterized by: Step 2.3 includes the following steps: S2.3.1: Build a residual group consisting of 3 residual blocks connected in series. Each residual block consists of a Conv convolution with a kernel size of 3×3 and a stride of 1, a ReLU activation function, a Conv convolution with a kernel size of 3×3 and a stride of 1, and a residual connection. S2.3.2: Build the enhancement module. First, perform a ConvTranspose2d deconvolution operation on the input Input with a convolution kernel size of 4×4, a step size of 2, and a padding of 1. Then perform 6 residual group operations in sequence to obtain an enhanced feature map H with unchanged resolution and number of channels. The enhancement process of the enhanced feature map H is expressed as: In the above formula, H θ It represents the feature map output after the input feature map Input passes through the first residual block; H 2θ Represents the input feature map H θ The feature map output after the second residual block; H 3θ Represents the input feature map H 2θ The feature map output after the third residual block; Relu(Conv(H θ )) represents the input feature map H θ After the convolution operation, the Relu activation function operation is performed; Conv(Relu(Conv(H θ ))) represents the input feature map H θ After the convolution operation, the Relu activation function operation is performed and the convolution operation is performed at the end; Conv(H 2θ ) represents the input feature map H 2θ Perform convolution operation; Relu(Conv(H 2θ )) represents the input feature map H 2θ After the convolution operation, the Relu activation function operation is performed; Conv(Relu(Conv(H 2θ ))) represents the input feature map H 2θ After the convolution operation, the Relu activation function operation is performed and the convolution operation is performed at the end; ConvTranspose2d (H 3θ ) represents the input feature map H 3θ Perform deconvolution operation.

6. The image dehazing method based on multi-depth fusion of conditional generative adversarial networks according to claim 2 is characterized by: Step 2.4 includes the following steps: S2.4.1: Build an entry module for processing foggy images as input to the generator; Build a convolution operation Conv with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, followed by a layer of ReLU activation function. After the foggy image input to the generator G passes through this entry module, the output is a foggy feature map X with unchanged spatial resolution and 16 channels. S2.4.2: Build a downsampling encoder for learning haze features; S2.4.3: Add an enhancement module to enhance the low-resolution high-semantic feature representation capability after the fifth encoder layer to obtain the enhanced feature map X5′; S2.4.4: Build an upsampling decoder for reconstructing the dehazed image; S2.4.5: Build the network layer that outputs the results of the generator.

7. The image dehazing method based on multi-depth fusion of conditional generative adversarial networks according to claim 6 is characterized by: S2.4.2 includes the following steps: S2.4.2.1: construct a module composed of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2, which are added after parallel operation, to form the first layer of the downsampling encoder. This module outputs the haze image feature map X1 with unchanged resolution. Similarly, stack this module four times in sequence to form the second, third, fourth, and fifth layers of the downsampling encoder. S2.4.2.2: Create a downsampling operation between adjacent downsampling encoder layers, specifically, perform a convolution Conv operation with a kernel size of 3×3, a stride of 2, and a padding of 0 on the feature map output by the encoder layer of the previous layer. Therefore, the second, third, fourth, and fifth layers of the downsampling encoder output image feature maps with half the resolution and double the number of channels, denoted as X2, X3, X4, and X5 respectively; S2.4.3: An enhancement module is added after the fifth encoder layer to enhance the low-resolution high-semantic feature representation capability, and the enhanced feature map X5′ is obtained.

8. The image dehazing method based on multi-depth fusion of conditional generative adversarial networks according to claim 6 is characterized by: S2.4.4 includes the following steps: S2.4.4.1: A module composed of five parallel operations of the haze learning unit HLB constructed in S2.1 and the multi-depth projection module MDPM constructed in S2.2 is stacked in sequence to form the fifth, fourth, third, second, and first layers of the upsampling decoder of the generative network; S2.4.4.2: An upsampling operation is constructed between adjacent upsampling decoder levels. Specifically, a transposed convolution operation with a kernel size of 2×2 and a stride of 2 is performed, which doubles the resolution of the input feature map and halves the number of channels. S2.4.4.3: Let X5′ be the input of the fifth layer of the upsampling decoder. After being processed by the fifth layer of the upsampling decoder, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y4. S2.4.4.4: Let the feature map of Y4 and X4 be the input of the fourth layer of the upsampling decoder after bitwise addition. After processing by this block, the feature map with the same resolution as the input is output. After one upsampling operation, the feature map with doubled resolution and halved number of channels is obtained, which is recorded as Y3. S2.4.4.5: Let the feature map obtained by bitwise addition of Y3 and X3 be the input of the third layer of the upsampling decoder; after being processed by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y2; S2.4.4.6: Let the feature map of Y2 and X2 be the input of the third layer of the upsampling decoder. After being processed by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map has a doubled resolution and half the number of channels, which is recorded as Y1. S2.4.4.7: Let the feature map formed by bitwise addition of Y1 and X1 be used as the input of the first layer of the upsampling decoder; after processing by this block, the output is a feature map with the same resolution as the input. After one upsampling operation, the feature map is obtained with unchanged resolution and number of channels, which is denoted as Y.

9. The image dehazing method based on multi-depth fusion of conditional generative adversarial networks according to claim 2, characterized in that: Step 2.5 includes the following steps: S2.5.1: Build the entry module of the discriminator: Build a concat splicing function, a convolution operation Conv with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a LeakyReLU activation function. After the real clear image and the defogging generated image of the input discriminator D pass through this entry module, the output spatial resolution is halved and the number of channels is 64. S2.5.2: Build three main discriminator modules in series. Each module consists of a convolution operation Conv with a kernel size of 4×4, a stride of 2, and a padding of 1, a batch normalization layer, and a LeakyReLU activation function layer. After the feature map passes through this module, the output spatial resolution is halved and the number of channels is doubled. Finally, the discriminator outputs a discriminant matrix with a spatial resolution of 30×30 and a channel number of 1; The mean of the discriminator output matrix is ​​used as the True / False output; The output value 1 indicates that the image input to the discriminator is a truly collected haze-free image, and the output value 0 indicates that the image input to the discriminator is a haze-free image generated by the generator.

10. The image dehazing method based on multi-depth fusion of conditional generative adversarial networks according to claim 1, characterized in that: The step 3 comprises the following steps: S3.1: Initialize the hyperparameters required for training the model, the batch size, training rounds, and initial learning rate of the training set; Start training the model. Input the haze image to be defogged into the generator in MDF-GAN. Each time, the defogged image generated by the generator and its corresponding real haze-free image are input into the discriminator in the adversarial network. The discriminator will output 0 or 1, and repeat this adversarial generation learning process. S3.2: Construct a loss function for MDF-GAN to make the detailed information of the image generated by the generator more complete. The specific expression of the generator loss function using the multi-loss fusion scheme is as follows: Where, L gen represents the total loss of the generator; L con Indicates content loss; L P Indicates perceived loss; L adv Represents the loss of resistance; K m,i,j represents a real haze-free image with m channels and i×j size; m,i,j Represents a dehazed image with m channels and i×j size; It represents the feature map output by a real fog-free image with m channels and size i×j after passing through n layers of convolutional neural network; represents the feature map of the defogging image with m channels and size i×j after passing through n layers of convolutional neural network; D(x,y) represents the result obtained after the foggy image and the defogging image are input to the discriminator; D(x,K) represents the result obtained after the foggy image and the real fog-free image are input to the discriminator; E x,y [D(x,y)] represents the expected value of the result obtained after the foggy image and the defogged image are input into the discriminator in the joint distribution samples of the foggy image and the defogged image; E x,K [D(x,K)] represents the expected value of the result obtained after the foggy image and the real fog-free image are input into the discriminator in the joint distribution samples of the foggy image and the real fog-free image; E[*] represents the expected value of the sample distribution, ||*||1 represents the L1 norm, x represents the foggy image input to the generator, y represents the image after the generator completes the defogging, K represents the real fog-free image for comparison, C, W, H represent the channel, width and height of the feature map respectively, represents the nth layer feature map of the convolutional neural network, D(*) represents the discriminator, and D∈1-Lipschitz is the limiting condition of the discriminator parameters; The task of the discriminator is to distinguish the real haze-free image K from the haze-free image y generated by the generator, that is, to output 1 for the real haze-free image K and 0 for the haze-free image y generated by the generator; Therefore, the loss function of the discriminator uses the cross entropy binary classification loss function, denoted by L D , the specific calculation process is as follows: In the formula, ω n is the weight; K n Represents the probability of this type of output, y n represents the true value of the class; l1,…,l N They represent the N loss values ​​calculated after N sets of real haze-free images and corresponding dehazed images input into the discriminator; T represents the transpose of the vector.

Images