An image dehazing method based on adversarial network and multi-scale dense feature fusion

Abstract

The present invention discloses an image defogging method based on an adversarial network and multi-scale dense feature fusion, belonging to the field of image processing technology. The defogging method has the following specific steps: (1) constructing a conditional adversarial generative network for image defogging; (2) using a public image defogging data set to train the network until convergence; (3) using the trained generator network as an image defogging network, inputting a foggy image, and outputting a defogged image; the present invention introduces a residual dense block and a dense feature fusion module based on back-projection technology into the generator network, and uses them to perform local feature fusion and multi-scale dense feature fusion respectively, and gradually obtains a clearer image by effectively fusing features at different levels; thereby making the image defogging more natural and eliminating noise.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method based on an adversarial network and multi-scale dense feature fusion. Background Art

[0002] After searching, Chinese patent number CN102930514A discloses a fast image defogging method based on an atmospheric physical scattering model. Although the invention can perform image defogging through an atmospheric physical scattering model, the defogging is unnatural and there is noise. In recent years, the intensity of the light emitted by the scene will be attenuated when it passes through the haze and reaches the photosensitive element of the camera. In addition, the scattering effect of the haze on the light has greatly affected the imaging effect of the camera. Computer vision technology based on image analysis has been widely used in military defense, artificial intelligence, automatic control and other fields. Therefore, effective image defogging technology is needed as a pre-processing process for computer vision applications to reduce or eliminate the impact of haze. In the haze image imaging model, the atmospheric light intensity, the transmittance of the scene, and the image to be restored are all unknown quantities, and the only known thing is the haze image acquired by the camera. Therefore, the single image defogging problem is an ill-posed problem in mathematics. The key problem of image defogging is to obtain the transmittance image of the haze image, and the transmittance is a function of the image depth. In order to solve this ill-posed problem, the image defogging methods are mainly divided into defogging methods based on prior knowledge and defogging methods based on deep learning. These methods basically adopt a general network structure, but these structures have some limitations; therefore, it is particularly important to invent an image defogging method based on adversarial networks and multi-scale dense feature fusion;

[0003] Most of the existing image dehazing methods adopt general network structures. These structures often only utilize low-level local features of relatively shallow networks. However, high-level semantic information is difficult to be encoded by low-level features under limited receptive fields, which easily leads to unnatural and noisy image dehazing. To this end, we propose an image dehazing method based on adversarial networks and multi-scale dense feature fusion. Summary of the invention

[0004] The purpose of the present invention is to solve the defects in the prior art and to propose an image defogging method based on an adversarial network and multi-scale dense feature fusion.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for image defogging based on adversarial network and multi-scale dense feature fusion. The specific steps of the defogging method are as follows:

[0007] (1) Construct a conditional adversarial generation network for image dehazing;

[0008] (2) Use a public image dehazing dataset to train the network until convergence;

[0009] (3) The trained generator network is used as the image dehazing network, with the input being the foggy image and the output being the dehazed image.

[0010] Furthermore, the conditional adversarial generative network in step (1) includes a generator network, a discriminator network and an overall objective function.

[0011] Furthermore, the structural design of the generator network is based on the U-Net structure, which includes three parts: an encoding module, a decoding module and a feature recovery module. The encoding module is provided with an encoder and integrated with a residual dense block RDB module and a DFF module based on back projection technology. The decoding module is provided with a decoder; the specific operation process of the generator network is as follows:

[0012] S1: Set the input of the network to;

[0013] S2: The foggy image is sent to the encoder part, and after the convolution operation, the image is downsampled as a feature map;

[0014] S3: In each downsampling stage, the feature map is sent to a residual dense block RDB module for local dense feature extraction and local feature fusion, and then the fused feature map is downsampled through a convolutional layer with a step size of 2. The downsampled feature map is then sent to the DFF module to make up for the missing spatial information;

[0015] S4: The number of feature maps is doubled at each stage, and the feature maps after 4 downsampling are sent to the feature recovery module;

[0016] S5: The feature map enhanced by the feature recovery module is upsampled to the original input size through 4 deconvolutions;

[0017] S6: After each upsampling, the output is connected with the potential features from the corresponding layer of the encoder, and then input into the residual dense block RDB module for feature refinement;

[0018] S7: The refined features are then input into the DFF module for adaptive feature fusion.

[0019] Furthermore, the structure of the discriminator network adopts PatchGAN, and its output is an N*N matrix.

[0020] Furthermore, the overall objective function adopts the WGAN-GP loss and is modified to be conditionally set as the adversarial training loss, as shown below:

[0021]

[0022] Introducing L1 and L2 losses into the above formula yields:

[0023] L1(G)=E I,J [∣∣JG(I)∣∣1] (2)

[0024] L2(G)=E I,J [∣∣JG(I)∣∣2] 2 (3)

[0025] Combining the above losses, we get the final overall objective function:

[0026]

[0027] Where: G is the generator network structure, D is the discriminator network structure, I is the foggy image, J is the real image, is the straight line sampling between J and the image G(I) generated by the generator network structure, λ GP is the weight factor.

[0028] Furthermore, the feature fusion adopts a DFF module of a back-projection technique, and the back-projection technique is used to generate high-resolution content by minimizing the reconstruction error between the estimated high-resolution result and multiple observed low-resolution inputs.

[0029] Furthermore, the feature recovery module includes 8 residual dense block RDB modules, and each residual dense block RDB module includes a dense connection layer, local feature fusion and local residual learning, and its specific operation process is as follows:

[0030] SS1: reads the state of the previous RDB module through a continuous memory mechanism and then passes it to each layer of the current RDB module, while making full use of all layers through local dense connections;

[0031] SS2: Local feature fusion is performed through a 1*1 convolution layer to adaptively retain the accumulated features;

[0032] SS3: Further improve information flow and enhance the representation capability of the network through local residual learning.

[0033] Furthermore, the encoder has a DFF module at the nth level. is defined as:

[0034]

[0035] Where: i n is the latent feature of the encoder at level n, It is an enhanced feature achieved through feature fusion. It is the enhanced features of all the previous (n-1) stages of the encoder that have been fused by the DFF module;

[0036] The decoder is in the DFF module at the nth level is defined as:

[0037]

[0038] Where: j n is the latent feature of the encoder at level n, is the enhanced feature achieved by feature fusion, L is the number of feature levels, It is the enhanced features of all the (n-1) stages before the decoder that have been fused by the DFF module.

[0039] Furthermore, the enhanced features are updated in a progressive manner, and the specific updating steps are as follows:

[0040] SSS1: The feedback error between the latent features and the enhanced features of the DDF module at the tth feature fusion is calculated by formula 1;

[0041] SSS2: The feedback error is downsampled to the same scale as the enhanced feature through formula 2; then it is added to the latent feature to obtain the fused enhanced feature;

[0042] The formula 1 in step SSS1 is as follows:

[0043]

[0044] The formula 2 described in step SSS2 is as follows:

[0045]

[0046] Where: is the feedback error, To enhance the features, is the upsampling operator, For potential characteristics, is the downsampling operator.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The image defogging method based on conditional generative adversarial network and multi-scale dense feature fusion introduces residual dense blocks and dense feature fusion modules based on back-projection technology into the generator network, and uses them to perform local feature fusion and multi-scale dense feature fusion respectively, and gradually obtains clearer images by effectively fusing features at different levels; and an RDB module is applied at each level of the encoding and decoding part. While extracting rich local features through densely connected convolutional layers, more effective features can be adaptively learned from previous and current local features through local feature fusion, which is conducive to making image defogging more natural and eliminating noise;

[0049] 2. The structural design of the generator network in the image dehazing method based on conditional generative adversarial network and multi-scale dense feature fusion is based on the U-Net structure. There are some inherent problems when using the U-Net structure to process image dehazing, such as downsampling in the encoder and loss of spatial information, and lack of sufficient connection between non-adjacent layer features. Therefore, the present invention introduces a DFF module based on the back projection technology at each level of the generator network, and performs multi-scale dense feature fusion to make full use of features of different scales. This is conducive to avoiding the problem of loss of spatial information in fully extracted local dense features. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0051] Figure 1 This is an overall flow chart of an image defogging method based on an adversarial network and multi-scale dense feature fusion proposed by the present invention;

[0052] Figure 2 Schematic diagram of the generator network structure and the discriminator network structure of the present invention;

[0053] Figure 3 Schematic diagram of the residual dense block RDB module of the present invention;

[0054] Figure 4 FIG. 4 is a schematic diagram of a network architecture of a DFF module at the nth level of an encoder according to the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0056] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0057] Reference Figure 1-4 , an image defogging method based on adversarial network and multi-scale dense feature fusion, the specific steps of the defogging method are as follows:

[0058] (1) Construct a conditional adversarial generation network for image dehazing;

[0059] (2) Use a public image dehazing dataset to train the network until convergence;

[0060] (3) The trained generator network is used as the image dehazing network, with the input being the foggy image and the output being the dehazed image.

[0061] Step (1) The conditional adversarial generation network includes a generator network, a discriminator network and an overall objective function.

[0062] The structural design of the generator network is based on the U-Net structure, which consists of three parts: the encoding module, the decoding module and the feature recovery module. The encoding module is provided with an encoder and integrates a residual dense block RDB module and a DFF module based on back-projection technology. The decoding module is provided with a decoder. The specific operation process of the generator network is as follows:

[0063] S1: Set the input of the network to;

[0064] S2: The foggy image is sent to the encoder part, and after the convolution operation, the image is downsampled as a feature map;

[0065] S3: In each downsampling stage, the feature map is sent to a residual dense block RDB module for local dense feature extraction and local feature fusion, and then the fused feature map is downsampled through a convolutional layer with a step size of 2. The downsampled feature map is then sent to the DFF module to make up for the missing spatial information;

[0066] S4: The number of feature maps is doubled at each stage, and the feature maps after 4 downsampling are sent to the feature recovery module;

[0067] S5: The feature map enhanced by the feature recovery module is upsampled to the original input size through 4 deconvolutions;

[0068] S6: After each upsampling, the output is connected with the potential features from the corresponding layer of the encoder, and then input into the residual dense block RDB module for feature refinement;

[0069] S7: The refined features are then input into the DFF module for adaptive feature fusion.

[0070] The structure of the discriminator network adopts PatchGAN, and its output is an N*N matrix.

[0071] The overall objective function adopts the WGAN-GP loss and modifies it to be conditioned as the adversarial training loss as follows:

[0072]

[0073] Introducing L1 and L2 losses into the above formula yields:

[0074] L1(G)=E I,J [∣∣JG(I)∣∣1] (2)

[0075] L2(G)=E I,J [∣∣JG(I)∣∣2] 2 (3)

[0076] Combining the above losses, we get the final overall objective function:

[0077]

[0078] Where: G is the generator network structure, D is the discriminator network structure, I is the foggy image, J is the real image, is the straight line sampling between J and the image G(I) generated by the generator network structure, λ GP is the weight factor.

[0079] Feature fusion uses the DFF module with back-projection technique, which is used to generate high-resolution content by minimizing the reconstruction error between the estimated high-resolution result and multiple observed low-resolution inputs.

[0080] The feature recovery module contains 8 residual dense block RDB modules, and each residual dense block RDB module includes a dense connection layer, local feature fusion and local residual learning. The specific operation process is as follows:

[0081] SS1: reads the state of the previous RDB module through a continuous memory mechanism and then passes it to each layer of the current RDB module, while making full use of all layers through local dense connections;

[0082] SS2: Local feature fusion is performed through a 1*1 convolution layer to adaptively retain the accumulated features;

[0083] SS3: Further improve information flow and enhance the representation capability of the network through local residual learning.

[0084] Encoder DFF module at level n is defined as:

[0085]

[0086] Where: i n is the latent feature of the encoder at level n, It is an enhanced feature achieved through feature fusion. It is the enhanced features of all the previous (n-1) stages of the encoder that have been fused by the DFF module;

[0087] The decoder is at the DFF module at level n is defined as:

[0088]

[0089] Where: j n is the latent feature of the encoder at level n, is the enhanced feature achieved by feature fusion, L is the number of feature levels, It is the enhanced features of all the (n-1) stages before the decoder that have been fused by the DFF module.

[0090] The enhanced features are updated in a progressive manner. The specific update steps are as follows:

[0091] SSS1: The feedback error between the latent features and the enhanced features of the DDF module at the tth feature fusion is calculated by formula 1;

[0092] SSS2: The feedback error is downsampled to the same scale as the enhanced feature through formula 2; then it is added to the latent feature to obtain the fused enhanced feature;

[0093] Step SSS1 formula 1 is as follows:

[0094]

[0095] Step SSS2 formula 2 is as follows:

[0096]

[0097] Where: is the feedback error, To enhance the features, is the upsampling operator, For potential characteristics, is the downsampling operator.

[0098] The working principle and use process of the present invention are as follows: the image defogging method based on conditional generative adversarial network and multi-scale dense feature fusion first needs to construct a conditional adversarial generative network for image defogging; then use a public image defogging data set to train the network until convergence; finally, the trained generator network is used as the image defogging network, the input is a foggy image, and the output is a defogged image; the specific operation process of the generator network is as follows: first, the input of the network is set to; then the foggy image is sent to the encoder part, and after the convolution operation, the image is downsampled as a feature map; then, in each downsampling stage, the feature map is sent to a residual dense block RDB module for local dense feature extraction and local feature fusion, and then the fused feature map is downsampled through a convolution layer with a step size of 2, and then the downsampled feature map is sent to the DF F module, to make up for the missing spatial information; then the number of feature maps in each stage is doubled, and the feature maps after 4 downsampling are sent to the feature recovery module; then the feature maps enhanced by the feature recovery module are upsampled to the original input size by 4 deconvolutions; then after each upsampling, the output is connected with the potential features from the corresponding layer of the encoder, and then input together into the residual dense block RDB module for feature refinement; finally, the refined features are input into the DFF module for adaptive feature fusion; the present invention introduces residual dense blocks and dense feature fusion modules based on back-projection technology into the generator network, and utilizes them to perform local feature fusion and multi-scale dense feature fusion respectively, and gradually obtains clearer images by effectively fusing features at different levels; thereby making image defogging more natural and eliminating noise.

[0099] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An image dehazing method based on adversarial network and multi-scale dense feature fusion, characterized in that: The specific steps of the defogging method are as follows: (1) Construct a conditional adversarial generation network for image dehazing; (2) Use a public image dehazing dataset to train the network until convergence; (3) The trained generator network is used as the image defogging network, with the input being the foggy image and the output being the defogged image; The conditional adversarial generation network includes a generator network, a discriminator network and an overall objective function; The structural design of the generator network is based on the U-Net structure, which consists of three parts: an encoding module, a decoding module and a feature recovery module. The encoding module is provided with an encoder and integrates a residual dense block RDB module and a DFF module based on back-projection technology. The decoding module is provided with a decoder; the specific operation process of the generator network is as follows: S1: Set the input of the network to; S2: The foggy image is sent to the encoder part, and after the convolution operation, the image is downsampled as a feature map; S3: In each downsampling stage, the feature map is sent to a residual dense block RDB module for local dense feature extraction and local feature fusion, and then the fused feature map is downsampled through a convolutional layer with a step size of 2. The downsampled feature map is then sent to the DFF module to make up for the missing spatial information; S4: The number of feature maps is doubled at each stage, and the feature maps after 4 downsampling are sent to the feature recovery module; S5: The feature map enhanced by the feature recovery module is upsampled to the original input size through 4 deconvolutions; S6: After each upsampling, the output is connected with the potential features from the corresponding layer of the encoder, and then input into the residual dense block RDB module for feature refinement; S7: The refined features are then input into the DFF module for adaptive feature fusion; The structure of the discriminator network adopts PatchGAN, and its output is an N*N matrix; The feature recovery module includes 8 residual dense block RDB modules, and each residual dense block RDB module includes a dense connection layer, local feature fusion and local residual learning. The specific operation process is as follows: SS1: reads the state of the previous RDB module through a continuous memory mechanism and then passes it to each layer of the current RDB module, while making full use of all layers through local dense connections; SS2: Local feature fusion is performed through a 1*1 convolution layer to adaptively retain the accumulated features; SS3: Further improve information flow and enhance the representation capability of the network through local residual learning.

2. The image defogging method based on adversarial network and multi-scale dense feature fusion according to claim 1 is characterized in that: The overall objective function adopts the WGAN-GP loss and modifies it to be conditioned as the adversarial training loss as follows: Introducing L1 and L2 losses into the above formula yields: L1(G)=E I，J [||J-G(I)||1] (2) L2(G)=E I，J [||JG(I)||2] 2 (3) Combining the above losses, we get the final overall objective function: Where: G is the generator network structure, D is the discriminator network structure, I is the foggy image, J is the real image, is the straight line sampling between J and the image G(I) generated by the generator network structure, λ GP is the weight factor.

3. The image defogging method based on adversarial network and multi-scale dense feature fusion according to claim 1, characterized in that: The feature fusion adopts a DFF module of a back-projection technique, which is used to generate high-resolution content by minimizing the reconstruction error between an estimated high-resolution result and multiple observed low-resolution inputs.

4. The image defogging method based on adversarial network and multi-scale dense feature fusion according to claim 1, characterized in that: The encoder is in the DFF module at the nth level is defined as: Where: i n is the latent feature of the encoder at level n, It is an enhanced feature achieved through feature fusion. It is the enhanced features of all the previous (n-1) stages of the encoder that have been fused by the DFF module; The decoder is in the DFF module at the nth level is defined as: Where: j n is the latent feature of the encoder at level n, is the enhanced feature achieved by feature fusion, L is the number of feature levels, It is the enhanced features of all the (n-1) stages before the decoder that have been fused by the DFF module.

5. The image defogging method based on adversarial network and multi-scale dense feature fusion according to claim 4, characterized in that: The enhanced features are updated in a progressive manner, and the specific updating steps are as follows: SSS1: The feedback error between the latent features and the enhanced features of the DDF module at the tth feature fusion is calculated by formula 1; SSS2: The feedback error is downsampled to the same scale as the enhanced feature through formula 2; then it is added to the latent feature to obtain the fused enhanced feature; The formula 1 in step SSS1 is as follows: The formula 2 described in step SSS2 is as follows: Where: is the feedback error, To enhance the features, is the upsampling operator, For potential characteristics, is the downsampling operator.

Images