Image defogging method based on generative adversarial network

Abstract

The present application relates to the field of image processing, and more particularly to an image defogging method based on a generative adversarial network. In the first stage, the generator of the generative adversarial network is trained by synthetic fog data, and the sum of the smooth L1 loss between the defogging image of the synthetic fog paired image and the fog-free image and the smooth L1 loss between the reconstructed fog image and the synthetic fog paired image is taken as the total loss function of the first stage; in the second stage, real fog data is used for training, the spatial position contrast loss of the feature layer of the real fog image and the feature layer of the defogging image of the real fog image is calculated, and a multi-layer contrast loss is constructed; the style transfer loss is constructed based on the defogging image features of the real fog image generated by the generator of the two stages; the identity consistency loss is constructed based on the L1 norm loss of the real fog-free image and the defogging image of the real fog-free image; the total loss function of the second stage is constructed for training, and the target generative adversarial network is obtained. The present application improves the efficiency and accuracy of image defogging.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image defogging method based on a generative adversarial network. BACKGROUND

[0002] The research and development of image defogging technology aims to solve the problem of weakened visual information caused by fog and haze. The scattering effect of tiny water droplets in the fog on light will cause damage to image acquisition, resulting in loss of information and reduction of brightness, contrast and overall visibility. These factors have a serious impact on the efficiency of image processing systems. The core goal of image defogging technology is to restore a clear and visible image from a fog-affected image. In safety-sensitive application fields, such as intelligent transportation systems, extreme weather conditions, especially foggy weather, significantly reduce the driver's visual judgment ability through the window to the road conditions in front, increasing the potential risk of driving. Image defogging technology not only improves image quality, but more importantly, ensures the accuracy and usability of image information, thereby supporting related systems to make effective decisions.

[0003] Traditional image defogging methods mainly rely on non-physical models and use image enhancement techniques (such as histogram equalization and local contrast enhancement) to visually improve fog-affected images. However, these methods often ignore the physical formation process of fog, especially without considering the concentration, distribution of fog and the complex interaction with ambient light, resulting in visual improvement but not fundamentally restoring the true scene of the image, sometimes causing loss of image details or distortion of colors.

[0004] In recent years, defogging algorithms based on physical models have begun to be researched and applied. These algorithms attempt to establish mathematical models about atmospheric scattering and ambient light to infer and restore the true state of the haze-free image. For example, the atmospheric scattering model assumes that fog is caused by tiny water droplets or ice crystals in the air affecting light propagation through scattering. By estimating the ambient light and scattering parameters, this model can be used to restore the color and contrast of the image to its original state.

[0005] The introduction of deep learning techniques has brought revolutionary progress to the field of image dehazing. By constructing deep neural networks, models can be trained to automatically learn the non-linear mapping relationship between hazy images and clear images. This method utilizes the power of big data, through large-scale image dataset training, the network can recognize and simulate the impact of fog, achieving more accurate image restoration. Some scholars in the existing technology use innovative end-to-end learning methods to directly learn and estimate the mapping relationship between the hazy image block and its transmission graph from the hazy image. This method uses deep convolutional neural network structure to automatically extract image features and generate dehazed images by learning a large number of hazy and haze-free image pairs, greatly simplifying the complex preprocessing and parameter adjustment process in traditional dehazing technology. There is also existing technology that proposes a multi-scale convolutional neural network (MSCNN) to improve the image dehazing effect by introducing a multi-scale structure. The network design includes multiple scale processing units, each unit is responsible for capturing image details at different scales, and the image quality is optimized layer by layer from coarse to fine. This hierarchical processing method makes MSCNN more effective in dealing with different degrees of haze impact while preserving more image details.

[0006] In addition, the use of generative adversarial networks (GAN) also shows great potential in improving the naturalness and detail performance of dehazed images. In this framework, the generative network is responsible for generating dehazed images, while the adversarial network tries to distinguish between dehazed images and real haze-free images. Through this adversarial process, the dehazing model can generate more natural and delicate images. Existing technology proposes a conditional generative adversarial network (cGAN) that successfully removes haze by learning to map hazy images to their clear corresponding images. The Cycle-Dehaze network is also proposed, which improves the recovery of texture information by introducing cycle consistency and perceptual loss techniques. In addition, an innovative domain adaptive dehazing (DAD) method uses image transfer models to effectively convert simulated hazy images into realistic images, significantly improving the dehazing effect in real-world scenarios. Some scholars have also proposed a synthetic-to-real dehazing framework based on physical priors (PSD), which improves the method by using traditional priors or principles (such as dark channel) to guide the transition from synthetic to real, adapting the existing dehazing model for practical applications. In addition, the D4 unpaired dehazing framework is introduced, which enhances the training effect of the dehazing network by mining the scattering coefficient and depth information in hazy and clear images.

[0007] However, due to the difficulty in obtaining both foggy and non-foggy images of the same scene in real-world environments, especially in dynamic outdoor scenes, the coverage and quality of training data are limited. The scarcity and difficulty of collecting such data make the model training insufficient, making it difficult to adapt to the changing needs of practical applications. Therefore, many dehazing models rely on synthetic data for training, which creates fog effects by applying an atmospheric scattering model on clean images, but this method often does not match the simulated atmospheric light values and transmittance with real fog conditions, resulting in a large deviation between the dehazing effect in actual application and training. The differences in lighting and visual texture between synthetic fog images and real fog images make it difficult for the model to handle real complex fog scenes. In addition, although the generative adversarial network can generate visually convincing dehazed images, there are often problems of excessively high or low color saturation and unnatural texture details in the process of directly converting foggy images into non-foggy images. These distortion problems reduce the naturalness and practical value of the dehazed images, especially in application scenarios sensitive to color and texture details. SUMMARY

[0008] To this end, the technical problem to be solved by the present application is to overcome the problem of poor model precision caused by the difference between synthetic data and real scenes when existing methods train models using synthetic data, and the defects of color and texture distortion in the process of converting foggy images into non-foggy images.

[0009] To solve the above technical problems, the present application provides an image dehazing method based on a generative adversarial network, comprising the following steps:

[0010] An image data set is obtained, which includes different synthetic fog data and real fog data. The synthetic fog data includes a non-fog image and its corresponding synthetic fog paired image, and the real fog data includes a real fog image and its corresponding real non-fog image. The image data set is preprocessed, and the preprocessed image data set is divided into a training set and a validation set;

[0011] The synthetic fog paired image is input into the generator of the generative adversarial network to obtain a dehazed image of the synthetic fog paired image;

[0012] The dehazed image of the synthetic fog paired image is reconstructed to obtain a reconstructed fog image of the synthetic fog paired image;

[0013] The sum of the smooth L1 loss between the dehazed image of the synthetic fog paired image and the non-fog image, and the smooth L1 loss between the reconstructed fog image and the synthetic fog paired image, is taken as the total loss function of the first stage;

[0014] The generator is trained by the total loss function of the first stage to obtain a first-stage trained generator;

[0015] inputting the real fog image into the generator to obtain a feature map of the real fog image and a defogged image of the real fog image;

[0016] inputting the real fog-free image into the generator to obtain a defogged image of the real fog-free image;

[0017] inputting the defogged image of the real fog image into the post-feature extractor to obtain a defogged image feature of the real fog image;

[0018] obtaining a multi-layer feature of the real fog image by passing the feature map of the real fog image through a two-layer perception network;

[0019] obtaining a multi-layer defogged image feature of the real fog image by passing the defogged image feature of the real fog image through a two-layer perception network;

[0020] constructing a multi-layer contrast loss by calculating a contrast loss function of each spatial position in the defogged image feature layer of each feature layer of the real fog image and the same layer;

[0021] constructing a style transfer loss based on the first defogged image feature of the real fog image generated by the generator trained in the first stage and the second defogged image feature of the real fog image generated by the generator updated with the parameters in the second stage;

[0022] constructing an identity consistency loss based on an L1 norm loss between the real fog-free image and the defogged image of the real fog-free image;

[0023] weighting and summing the multi-layer contrast loss, the style transfer loss, the identity consistency loss, the generator loss and the discriminator loss to obtain a total loss function in the second stage;

[0024] training the generative adversarial network through the total loss function in the second stage, verifying the trained generative adversarial network through a verification set, and obtaining a target generative adversarial network;

[0025] inputting the to-be-defogged image into the target generative adversarial network to output a defogged image of the to-be-defogged image.

[0026] Preferably, the inputting the defogged image of the real fog image into the post-feature extractor to obtain the defogged image feature of the real fog image comprises:

[0027] passing the defogged image of the real fog image through a 3x3 convolution layer, and then passing the defogged image through twice 2-fold downsampling to obtain a low-resolution feature map of the real fog image;

[0028] passing the low-resolution feature map of the real fog image through three residual modules connected in sequence to obtain the defogged image feature of the real fog image.

[0029] Preferably, the formula of the total loss function of the second stage is:

[0030] L2=λ1L C+ +λ2L S +λ3L I +λ4(L G +L D ),

[0031] wherein L2 is the total loss function of the second stage, λ1 is the weight of the multi-layer contrast loss, L C is the multi-layer contrast loss, λ2 is the weight of the style transfer loss, L S is the style transfer loss, λ3 is the weight of the identity consistency loss, L I is the identity consistency loss, λ4 is the weight of the generator loss and the discriminator loss, L G is the generator loss, L D is the discriminator loss;

[0032] The formula of the multi-layer contrast loss L C is:

[0033]

[0034] wherein E x～X represents the mathematical expectation under the condition that the real fog image x is subject to the distribution X, X is the distribution of the real fog image in the sample space, L is the set number of layers, l is the selected layer number index, s is the spatial position index, l'(.) is the contrast loss function, is the feature vector of the sub-block with the spatial position s in the lth dehazing image feature layer of the real fog image, is the feature vector of the sub-block with the spatial position s in the lth feature layer of the real fog image, is the feature vector set of the sub-blocks except the sub-block with the spatial position s in the lth feature layer of the real fog image;

[0035] The formula of the style transfer loss L S is:

[0036]

[0037] wherein, is the second dehazing image feature of the real fog image generated by the generator after the second stage update parameter, is the first dehazing image feature of the real fog image generated by the generator trained in the first stage, and ||.||1 is the L1 norm;

[0038] The formula of the generator loss L G is:

[0039]

[0040] wherein E G(x)～r denotes the mathematical expectation under the condition that the dehazed image G(x) of the real fog image obeys distribution r, r is a distribution of the dehazed image generated by the generator in a sample space, D(.) is a discriminator of the generative adversarial network, and G(x) is the dehazed image of the real fog image;

[0041] The formula of the discriminator loss L D is as follows:

[0042] L D = E y～I [(D(y)-1) 2 ]+E G(x)～r [D(G(x)) 2 ],

[0043] wherein E y～I denotes the mathematical expectation under the condition that the real haze-free image y obeys distribution I, I is a distribution of the real haze-free image in a sample space;

[0044] The formula of the identity consistency loss L I is as follows:

[0045] L I = E y～I ∥G(y)-y∥1,

[0046] wherein E y～I denotes the mathematical expectation under the condition that the real fog image x obeys distribution I, G(x) is the dehazed image of the real fog image, x is the real fog image, and ∥.∥1 is an L1 norm.

[0047] Preferably, the generator of the generative adversarial network comprises: a pre-feature extractor and a dehazing network; wherein the dehazing network comprises: a plurality of dehazing modules connected in sequence, each dehazing module comprising: an image enhancement unit and a dehazing unit connected in sequence, and taking the dehazing feature map output by the last dehazing module as the dehazed image;

[0048] The input feature of the dehazing module is input into the dehazing module to output a dehazing feature map, comprising:

[0049] The input feature of the dehazing module is input into the image enhancement unit to perform feature enhancement, and an output enhanced feature map of the image feature enhancement unit is obtained;

[0050] The output enhanced feature map of the image feature enhancement unit is input into the dehazing unit to output a dehazing feature map, comprising:

[0051] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch to output an atmospheric light matrix;

[0052] The output enhanced feature map of the image feature enhancement unit is input into a transmission matrix estimation branch, and a transmission graph matrix is output;

[0053] Based on the atmospheric light matrix, the transmission graph matrix, the output enhanced feature map of the image feature enhancement unit, and the input feature of the defogging module, a defogging feature map is calculated.

[0054] Preferably, the input feature of the defogging module is enhanced by the image enhancement unit to obtain an output enhanced feature map of the image feature enhancement unit, including:

[0055] The input feature of the defogging module is dimensionally reduced to obtain a dimensionally reduced input feature of the defogging module;

[0056] The dimensionally reduced input feature of the defogging module is respectively input into three linear layers to obtain a key vector, a query vector, and a value vector;

[0057] Based on the key vector, the query vector, and the value vector, an attention weight is calculated.

[0058] The formula for calculating the attention weight is:

[0059]

[0060] wherein, O i is the output at the i-th pixel position, Q is the query vector, K is the key vector, V is the value vector, e is a natural constant,. T is the transpose, is the square root of the key vector channel dimension, i is the index of the query vector, j is the index of the value vector, k is the index of the key vector, N is HxW, H is the height of the input feature of the defogging module, and W is the width of the input feature of the defogging module.

[0061] The attention weight is dimensionally converted and then up-sampled to obtain a dimensionally converted and up-sampled attention weight.

[0062] The dimensionally converted and up-sampled attention weight is multiplied by a learnable parameter and then added to the input feature of the defogging module to obtain an output enhanced feature map of the image feature enhancement unit.

[0063] The formula for multiplying the dimensionally converted and up-sampled attention weight by the learnable parameter and then adding it to the input feature of the defogging module to obtain the output enhanced feature map of the image feature enhancement unit is:

[0064]

[0065] wherein, The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0066] Preferably, the output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0067] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0068] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0069] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0070] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0071] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0072] Preferably, the output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0073] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0074] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0075] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0076] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0077] The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch, and an atmospheric light matrix is output.

[0078] Preferably, the output enhanced feature map of the image feature enhancement unit, the input feature of the defogging module, and the atmospheric light matrix and the transmission matrix are calculated to obtain the defogging feature map, and the calculation formula is:

[0079]

[0080] wherein, is the defogging feature map, is the output enhanced feature map of the image feature enhancement unit, and is an element-wise multiplication, is the transmission matrix, is the atmospheric light matrix, is the input feature of the defogging module.

[0081] Preferably, the defogging image of the synthetic fog paired image, the atmospheric light matrix output by the atmospheric scattering matrix estimation branch in the defogging unit of the last defogging module, and the transmission matrix output by the transmission matrix estimation branch in the defogging unit of the last defogging module are input into the image reconstruction module to obtain a reconstructed fog image, and the calculation formula of the reconstructed fog image is:

[0082]

[0083] wherein, I rec is the reconstructed fog image, is the defogging image of the synthetic fog paired image, is an element-wise multiplication, is the transmission matrix output by the transmission matrix estimation branch in the defogging unit of the last defogging module, is the atmospheric light matrix output by the atmospheric scattering matrix estimation branch in the defogging unit of the last defogging module, is the inverse matrix of , is the defogging feature map output by the last defogging module.

[0084] Preferably, the image to be defogged is input into the target generative adversarial network to output a defogging image of the image to be defogged, comprising:

[0085] The image to be defogged is input into the front feature extractor to extract a feature map of the image to be defogged, comprising:

[0086] After the image to be defogged is input into a 3*3 convolution layer, the image to be defogged is input into two 2 times down-sampling to obtain a low-resolution feature map of the image to be defogged;

[0087] After the low-resolution feature map of the image to be defogged is input into three residual modules connected in sequence, the low-resolution feature map of the image to be defogged is input into two 2 times up-sampling to obtain a feature map of the image to be defogged;

[0088] The feature map of the image to be defogged is input into the defogging network to output a defogging image of the image to be defogged.

[0089] The above technical scheme of the present application has the following beneficial effects compared with the prior art:

[0090] The image defogging method based on the generative adversarial network provided by the present application effectively solves the difference between the synthetic data and the real scene by designing two training stages. In the first stage, the defogging network is trained by synthetic data. The generation process of the synthetic data is relatively simple, and the complexity of the data is lower compared with the real fog image. Training the defogging network using synthetic data in the first stage not only solves the problem of data scarcity, but also strengthens the target effectiveness, interpretability and preliminary generalization ability of the model. In the second stage, the generative adversarial network is trained by real data. This way makes the model learn the characteristics of fog in the real scene in depth, effectively makes up for the gap between synthetic data and real scene, improves the adaptability and accuracy of the model in the real environment, and can ensure that effective defogging can be achieved even in the case of limited real non-fog image data.

[0091] The present application constructs a total loss function of the second stage including a multi-layer contrast loss, a style transfer loss, a generator loss, a discriminator loss and an identity consistency loss in the loss function of the second stage of the present application, which avoids the problem of color and texture distortion. The multi-layer contrast loss is constructed by calculating the contrast loss function of each spatial position in the feature layer of the real fog image and the feature layer of the defogged image of the real fog image, which makes the model learn the subtle differences between the images before and after defogging in different feature layers and spatial positions, so as to more accurately adjust the image in the defogging process, avoid color and texture distortion, and improve the naturalness and accuracy of the defogged image. The style transfer loss is introduced by comparing the defogged image features of the real fog image obtained by the defogging network after the first stage and the second stage, which makes the model better utilize the knowledge learned in the first stage when processing real fog images, further optimizes the defogging effect, and reduces the color and texture distortion problems that may be caused by changes in the training stage. The identity consistency loss is introduced by calculating the L1 norm between the real non-fog image and the defogged image of the real non-fog image, which can improve the generalization ability of the defogging generator, so as to further make the generated defogged image maintain the color, structure and texture characteristics of the real fog image, and further improve the accuracy of the defogged image.

[0092] This invention also designs a dehazing network, comprising multiple sequentially connected dehazing modules. Each dehazing module includes a sequentially connected image enhancement unit and a dehazing unit. Through a self-attention mechanism, the image enhancement unit can capture the interrelationships between features at different locations in the image, improving the image dehazing capability. It can also better preserve the original details and structure of the image during the dehazing process, reducing image distortion caused by dehazing and improving the quality of the dehazed image. The dehazing unit outputs the atmospheric light matrix and transmission map matrix respectively through atmospheric scattering matrix estimation and transmission matrix estimation branches. These are combined with the enhanced feature map to calculate the dehazing feature map, achieving a deep fusion of the physical model and deep learning. This combination makes the dehazing process more consistent with real physical principles, thereby improving the accuracy and reliability of dehazing, making the generated dehazed image closer to the real fog-free state, and improving the model's accuracy. Attached Figure Description

[0093] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0094] Figure 1 This is a flowchart of the steps of an image dehazing method based on generative adversarial networks according to the present invention.

[0095] Figure 2 This is a schematic diagram of multi-layer contrast loss.

[0096] Figure 3 This is a structural diagram of a generative adversarial network.

[0097] Figure 4 This is a structural diagram of the defogging unit.

[0098] Figure 5 This involves generating comparison images of the dehazing effect using adversarial networks. Figure 5 (a) in the image is a foggy image. Figure 5 (b) in the image is the dehazing image. Figure 5 (c) in the image represents the label image. Detailed Implementation

[0099] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0100] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of an image dehazing method based on generative adversarial networks according to the present invention.

[0101] Embodiment 1 of the present invention provides an image dehazing method based on generative adversarial networks, comprising the following steps:

[0102] obtain an image dataset, the image dataset comprising: different synthetic fog data, real fog data, the synthetic fog data comprising a fog-free image and a corresponding synthetic fog paired image, the real fog data comprising a real fog image and a corresponding real fog-free image, pre-process the image dataset, divide the pre-processed image dataset into a training set and a validation set;

[0103] input the synthetic fog paired image into a generator of a generative adversarial network to obtain a dehazed image of the synthetic fog paired image;

[0104] reconstruct the dehazed image of the synthetic fog paired image to obtain a reconstructed fog image of the synthetic fog paired image;

[0105] take the sum of a smooth L1 loss between the dehazed image of the synthetic fog paired image and the fog-free image and a smooth L1 loss between the reconstructed fog image and the synthetic fog paired image as a total loss function of the first stage;

[0106] In this embodiment, specifically, the formula of the total loss function of the first stage is:

[0107]

[0108] wherein L1 is the total loss function of the first stage, is the smooth L1 loss between the dehazed image of the synthetic fog paired image and the fog-free image, is the smooth L1 loss between the reconstructed fog image and the synthetic fog paired image;

[0109] Taking the smooth L1 loss between the dehazed image of the synthetic fog paired image and the fog-free image as an example, the formula of the smooth L1 loss is:

[0110]

[0111] wherein n f is the total number of pixel points of the dehazed image of the synthetic fog paired image or the fog-free image, and σ is a pixel point index, is the value of the σth pixel point of the fog-free image y f , and is the value of the σth pixel point of the dehazed image of the synthetic fog paired image, and is the synthetic fog paired image.

[0112] Compared with the L2 loss, the smooth L1 loss adopted in this embodiment can prevent potential gradient explosion.

[0113] train the generator through the total loss function of the first stage to obtain a first-stage trained generator;

[0114] input the real fog image into the generator to obtain a feature map of the real fog image and a defogged image of the real fog image;

[0115] input the real fog-free image into the generator to obtain a defogged image of the real fog-free image;

[0116] input the defogged image of the real fog image into the post-feature extractor to obtain a defogged image feature of the real fog image;

[0117] pass the feature map of the real fog image through the two-layer perception network M to obtain L-layer features of the real fog image {f l} L ;

[0118] pass the defogged image feature of the real fog image through the two-layer perception network M to obtain L-layer defogged image features of the real fog image

[0119] construct a multi-layer contrast loss L by calculating a contrast loss function of each spatial position in the same layer of the defogged image feature layer of each feature layer of the real fog image C ;

[0120] As shown in Figure 2 , Figure 2 is a multi-layer contrast loss schematic diagram.

[0121] the set of spatial positions of each feature layer of the real fog image and the same layer of the defogged image feature layer is N l represents the total number of spatial positions of each layer, is the last spatial position of each layer;

[0122] In this embodiment, specifically, the formula of the multi-layer contrast loss L C is:

[0123]

[0124] wherein E x～X represents a mathematical expectation under the condition that the real fog image x is subject to the distribution X, X is the distribution of the real fog image in the sample space, L is the set number of layers, l is the selected layer index, s is the spatial position index, l' (.) is the contrast loss function, is the feature vector of the sub-block with the spatial position s in the lth defogged image feature layer of the real fog image, is the feature vector of the sub-block with the spatial position s in the lth feature layer of the real fog image, is the set of feature vectors of the sub-blocks other than the sub-block with the spatial position s in the lth feature layer of the real fog image;

[0125] In this embodiment, specifically, the obtaining process of the single contrast loss function is as follows:

[0126] A sub-block in the dehazed image y of the real fog image is randomly selected as an anchor point, a sub-block corresponding to the anchor point in the real fog image x is represented as a positive sample, and other sub-blocks in the real fog image x are represented as negative samples, and the relevance between the anchor point and the positive sample is maximized through the noise contrast estimation module.

[0127] The anchor point, the positive sample and the V negative samples are converted into feature vectors and represented as c, c + and c - , respectively. The single contrast loss function can be represented as a cross-entropy loss, and the formula is as follows:

[0128]

[0129] wherein l'(c, c + , c - ) is the single contrast loss function, exp(.) represents an exponential function with e as the base, d(c, c + ) represents the feature space distance between the anchor point and the positive sample, that is, the cosine similarity between the anchor point and the positive sample, τ is an adjustment factor, and d(c, c ) represents the feature space distance between the anchor point and the vth negative sample.

[0130] The multi-layer contrast loss is introduced, so that the model can learn the subtle differences between the images before and after dehazing in different feature layers and spatial positions, thereby more accurately adjusting the images in the dehazing process, avoiding distortion of colors and textures, and improving the naturalness and accuracy of the dehazed images.

[0131] The first dehazed image feature of the real fog image generated by the generator trained in the first stage and the second dehazed image feature of the real fog image generated by the generator updated in the second stage are used to construct the style transfer loss L S .

[0132] In this embodiment, specifically, the formula of the style transfer loss L S is as follows:

[0133]

[0134] wherein is the second dehazed image feature of the real fog image generated by the generator updated in the second stage, is the first dehazed image feature of the real fog image generated by the generator trained in the first stage, and ||.||1 is the L1 norm.

[0135] The style transfer loss is introduced, so that the model can better utilize the knowledge learned in the first stage when processing the real fog image, further optimize the defogging effect, and reduce the color and texture distortion problems caused by the change of the training stage.

[0136] The L1 norm loss between the defogging image based on the real fog-free image and the real fog-free image is used to construct the identity consistency loss L I ;

[0137] In this embodiment, specifically, the formula of the identity consistency loss L I is as follows:

[0138] L I =E y～I ∥G(y)-y∥1,

[0139] wherein E y～I represents the mathematical expectation under the condition that the real fog-free image y obeys the distribution I, G(y) is the defogging image of the real fog-free image, y is the real fog-free image, and ||.||1 is the L1 norm.

[0140] The multi-layer contrast loss, the style transfer loss, the identity consistency loss, the generator loss and the discriminator loss are weighted and summed to obtain the total loss function of the second stage;

[0141] In this embodiment, preferably, the formula of the total loss function of the second stage is as follows:

[0142] L2=λ1L C+ +λ2L S +λ3L I +λ4(L G +L D ),

[0143] wherein L2 is the total loss function of the second stage, λ1 is the weight of the multi-layer contrast loss, L C is the multi-layer contrast loss, λ2 is the weight of the style transfer loss, L S is the style transfer loss, λ3 is the weight of the identity consistency loss, L I is the identity consistency loss, λ4 is the weight of the generator loss and the discriminator loss, L G is the generator loss, and L D is the discriminator loss;

[0144] In this embodiment, specifically, the formula of the generator loss L G is as follows:

[0145] L G =E G(x)～r (D(G(x))-1) 2 ,

[0146] Among them, E G(x)～r Let G(x) be the expected value of the dehazed image G(x) of the real fog image, given that the dehazed image follows a distribution r, where r is the distribution of the dehazed image generated by the generator in the sample space, D(.) is the discriminator of the generative adversarial network, and G(x) is the dehazed image of the real fog image.

[0147] Discriminator loss L D The formula is:

[0148] L D =E y～I [(D(y)-1) 2 ]+E G(x)～r [D(G(x)) 2 ],

[0149] Among them, E y～I This represents the mathematical expectation given that the real haze-free image y follows a distribution I, where I is the distribution of the real haze-free image in the sample space.

[0150] The generative adversarial network is trained using the total loss function in the second stage, and the trained generative adversarial network is validated using a validation set to obtain the target generative adversarial network.

[0151] Input the image to be dehazed into the target generative adversarial network, and output the dehazed image of the image to be dehazed.

[0152] In this embodiment, preferably, as follows: Figure 3 As shown, Figure 3 To generate the structure diagram of the adversarial network.

[0153] A generative adversarial network is constructed. The generator of the generative adversarial network includes: a pre-feature extractor and a dehazing network. The dehazing network includes: multiple dehazing modules connected in sequence. Each dehazing module includes: an image enhancement unit and a dehazing unit connected in sequence. The dehazing feature map output by the last dehazing module is used as the dehazing image.

[0154] In this embodiment, preferably, the input features of the dehazing module are input into the dehazing module, and the dehazing feature map is output, including:

[0155] The input features of the dehazing module are passed through the image enhancement unit for feature enhancement, resulting in the output enhanced feature map of the image feature enhancement unit, including:

[0156] Input features of the dehazing module Dimensionality reduction is performed to obtain the input features of the dimensionality-reduced dehazing module. Where B is the batch size, C is the number of channels of the input feature of the dehazing module, H is the height of the input feature of the dehazing module, and W is the width of the input feature of the dehazing module;

[0157] The input feature of the dimension-reduced haze removal module is input into a key linear layer with a convolution kernel of to obtain a key vector K;

[0158] The input feature of the dimension-reduced haze removal module is input into a query linear layer with a convolution kernel of to obtain a query vector Q;

[0159] The input feature of the dimension-reduced haze removal module is input into a value linear layer with a convolution kernel of to obtain a value vector V;

[0160] The attention weight is calculated based on the key vector, the query vector and the value vector;

[0161] The formula for calculating the attention weight is:

[0162]

[0163] wherein, O i is the output at the i-th pixel position, Q is the query vector, K is the key vector, V is the value vector, e is a natural constant,. T is the transpose, is the square root of the channel dimension of the key vector, i is the index of the query vector, j is the index of the value vector, k is the index of the key vector, N is HxW, H is the height of the input feature of the haze removal module, and W is the width of the input feature of the haze removal module;

[0164] After the dimension conversion, the attention weight is up-sampled to obtain the attention weight after the dimension conversion and the up-sampling;

[0165] After the attention weight after the dimension conversion and the up-sampling is multiplied by a learnable parameter, the input feature of the haze removal module is added to obtain the output enhanced feature map of the image feature enhancement unit;

[0166] The formula for obtaining the output enhanced feature map of the image feature enhancement unit by multiplying the attention weight after the dimension conversion and the up-sampling by a learnable parameter and adding the input feature of the haze removal module is:

[0167]

[0168] wherein, is the output enhanced feature map of the image feature enhancement unit, γ is a learnable parameter, O is the attention weight after the dimension conversion and the up-sampling, and I0 is the input feature of the haze removal module.

[0169] The feature map extracted by the pre-feature extractor is input into the haze removal network based on the atmospheric scattering model, and the atmospheric scattering model is usually simplified into the following formula:

[0170] I(x) = J(x)T(x) + A(1 - T(x)),

[0171] where x is the position of the pixel point, T(x) is the medium transmission map, A is the global atmospheric light, I(x) is the pixel value of the foggy image at the pixel point x, and J(x) is the pixel value of the non-fog image at the pixel point x.

[0172] For each pixel point, the above formula is deformed to obtain a reconstruction model J(x) of the clean image:

[0173]

[0174] Let κ be a convolution kernel used for feature extraction, and the above formula is:

[0175]

[0176] where Θ represents a convolution operation, and represents Hadamard multiplication.

[0177] Then it can be represented as a matrix-vector:

[0178]

[0179] where, is the matrix representation of κ, is the matrix representation of J, is the matrix representation of I, is the matrix representation of is the matrix representation of A.

[0180] Let F1 be the front feature extractor equivalent to Similarly, let F2 be the feature representation of , the formula is further rewritten as:

[0181]

[0182] Let be the estimate of , let be the estimate of F2, be the feature representation of , and let be the feature representation of , the formula is rewritten as:

[0183]

[0184] Since the main features do not change before and after dehazing, the output dehazing feature map of the dehazing unit is represented as:

[0185]

[0186] According to the above reasoning process, a dehazing unit based on a physical model can be constructed, and the key to the dehazing effect lies in accurately estimating the global atmospheric light and the medium transmission map, and the global atmospheric light and the medium transmission map are derived from the output enhanced feature map of the image feature enhancement unit, so one branch of the dehazing unit is to estimate the atmospheric light matrix from the output enhanced feature map of the image feature enhancement unit Another branch is to estimate the transmission map matrix from the output enhanced feature map of the image feature enhancement unit

[0187] As shown in Figure 4 , a structure diagram of the dehazing unit is shown in Figure 4

[0188] The output enhanced feature map of the image feature enhancement unit is input into the dehazing unit, and the output dehazing feature map is output, including:

[0189] The output enhanced feature map of the image feature enhancement unit is input into the atmospheric scattering matrix estimation branch, and the atmospheric light matrix is output.

[0190] In this embodiment, preferably, the output enhanced feature map of the image feature enhancement unit is input into the atmospheric scattering matrix estimation branch, and the atmospheric light matrix is output, including:

[0191] The output enhanced feature map of the image feature enhancement unit is input into the atmospheric scattering matrix estimation branch, and the atmospheric light matrix is output.

[0192] The pooled enhanced feature map is sequentially processed through a 3x3 convolution layer and a ReLU activation function, and then spliced with the pooled enhanced feature map to obtain initial fusion features.

[0193] The initial fusion features are sequentially processed through a 3x3 convolution layer and a ReLU activation function, and then spliced with the initial fusion features to obtain secondary fusion features.

[0194] The secondary fusion features are sequentially processed through a 1x1 convolution layer and a Sigmoid activation function, and then element-wise added to the initial fusion features to obtain atmospheric light pre-generated features.

[0195] The atmospheric light pre-generated features are processed through upsampling to obtain the atmospheric light matrix.

[0196] ​Generally, the distribution of the atmospheric light value in the image feature space is relatively uniform, so some unimportant information can be ignored by pooling. However, in the actual process, due to the density and distribution of fog, the direction and intensity of the light source are different, and the atmospheric light value in some areas is not completely uniform. Therefore, the application adopts hybrid pooling, which retains significant features while capturing some average information to improve the comprehensiveness of the features. Then, through a dense residual connection, the atmospheric light matrix is finally output. The above series of processes can be represented by the following formula:

[0197]

[0198] wherein, is the enhanced feature map after pooling, MP(.) is the hybrid average pooling, concat(.) is the splicing operation in the feature dimension, ReLU(.) is the ReLU activation function, is the atmospheric light matrix, up(.) is the up-sampling operation, Sigmoid(.) is the Sigmoid activation function, conv 1×1 (.) is the 1x1 convolution layer, conv 3×3 (.) is the 3x3 convolution layer, is the initial fusion feature, is the secondary fusion feature.

[0199] The output enhanced feature map of the image feature enhancement unit is input into the transmission matrix estimation branch to output the transmission map matrix.

[0200] In this embodiment, preferably, the output enhanced feature map of the image feature enhancement unit is input into the transmission matrix estimation branch to output the transmission map matrix, comprising:

[0201] The output enhanced feature map of the image feature enhancement unit is input into the transmission matrix estimation branch to output the transmission map matrix.

[0202] The feature map processed by the 3x3 convolution layer is sequentially processed by the 3x3 convolution layer and the ReLU activation function, and then spliced with the feature map processed by the 3x3 convolution layer to obtain the first spliced feature.

[0203] The first spliced feature is sequentially processed by the 3x3 convolution layer and the ReLU activation function, and then spliced with the first spliced feature to obtain the second spliced feature.

[0204] The second spliced feature is sequentially processed by the 3x3 convolution layer and the ReLU activation function, and then spliced with the second spliced feature to obtain the third spliced feature.

[0205] The third spliced feature is sequentially processed by a 1*1 convolution layer and a Sigmoid activation function, and then is added to the feature map processed by a 3*3 convolution layer element by element to obtain a transmission graph matrix.

[0206] Since the transmission graph matrix is closely related to the transmittance, and the transmittance is mainly affected by the depth of field of the entire scene, we need to fully utilize the global and local information of the image, and use a 3*3 convolution layer to replace the mixed pooling of the atmospheric scattering matrix estimation branch.

[0207] The acquisition process of the transmission graph matrix is represented by the following formula:

[0208]

[0209] wherein, is the feature map processed by a 3*3 convolution layer, is the first spliced feature, is the second spliced feature, is the third spliced feature, is the transmission graph matrix.

[0210] Based on the atmospheric light matrix, the transmission graph matrix, the output enhanced feature map of the image feature enhancement unit, and the input feature of the defogging module, a defogging feature map is calculated.

[0211] In this embodiment, preferably, the defogging feature map is calculated based on the atmospheric light matrix, the transmission graph matrix, the output enhanced feature map of the image feature enhancement unit, and the input feature of the defogging module, and the calculation formula is:

[0212]

[0213] wherein, is the defogging feature map, is the output enhanced feature map of the image feature enhancement unit, and is element-wise multiplication, is the transmission graph matrix, is the atmospheric light matrix, is the input feature of the defogging module.

[0214] Since the in the atmospheric scattering model is equivalent to A, and is essentially obtained by inversely transforming T, the calculation formula of the reconstructed fog image is obtained.

[0215] Preferably, in the embodiment, the defogged image of the synthetic fog paired image, the atmospheric light matrix output by the atmospheric scattering matrix estimation branch in the defogging unit of the last defogging module, and the transmission graph matrix output by the transmission matrix estimation branch in the defogging unit of the last defogging module are input into an image reconstruction module to obtain a reconstructed fog image, and a calculation formula of the reconstructed fog image is:

[0216]

[0217] wherein I rec is the reconstructed fog image, is the defogged image of the synthetic fog paired image, is an element-wise multiplication, is the transmission graph matrix output by the transmission matrix estimation branch in the defogging unit of the last defogging module, is the atmospheric light matrix output by the atmospheric scattering matrix estimation branch in the defogging unit of the last defogging module, is an inverse matrix of is an inverse matrix of is the defogging feature map output by the last defogging module.

[0218] Preferably, in the embodiment, the defogging image of the target to-be-defogged image is output by inputting the target to-be-defogged image into a target generative adversarial network, and the target to-be-defogged image comprises the following steps:

[0219] The feature map of the target to-be-defogged image is extracted by inputting the target to-be-defogged image into a front feature extractor, and the target to-be-defogged image comprises the following steps:

[0220] After the target to-be-defogged image passes through a 3*3 convolution layer, a low-resolution feature map of the target to-be-defogged image is obtained by twice 2-fold downsampling.

[0221] After the low-resolution feature map of the target to-be-defogged image passes through three residual modules connected in sequence, a feature map of the target to-be-defogged image is obtained by twice 2-fold upsampling.

[0222] The feature map of the target to-be-defogged image is input into a defogging network, and the defogging image of the target to-be-defogged image is output.

[0223] The front feature extractor captures the local features of the to-be-dehazed image, such as edges, textures and the like, through a 3*3 convolutional layer, and then is subjected to twice 2 times down-sampling, so as to not only reduce the data dimension and the subsequent calculation amount, but also highlight the main features and avoid overfitting. The low-resolution feature map of the to-be-dehazed image is subjected to three residual modules connected in sequence, and the gradient problem is solved by means of a jump connection, so as to realize deep feature learning and improve the processing capability of the dehazing network for different fog interference. Twice 2 times up-sampling restores the low-resolution feature map of the to-be-dehazed image to the original resolution, so as to not only reserve the key features, but also restore the image structure, thereby providing a high-quality input for the dehazing network. These operations enhance the adaptability and generalization capability of the dehazing network to different scenes and fog degrees, and also simplify the learning task of the dehazing network, so that the dehazing network converges more quickly, the training process is optimized, the training efficiency is improved, and the training time is shortened.

[0224] By designing two training stages, the difference between the synthetic data and the real scene is effectively solved. In the first stage, the dehazing network is trained by synthetic data. The generation process of the synthetic data is relatively simple, and the complexity of the data is lower compared with the real fog image. Training the dehazing network using synthetic data in the first stage not only solves the problem of data scarcity, but also strengthens the target effectiveness, interpretability and preliminary generalization capability of the model. In the second stage, the generative adversarial network is trained by real data. This way makes the model learn the characteristics of fog in real scenes in depth, effectively makes up for the gap between synthetic data and real scenes, improves the adaptability and accuracy of the model in real environment, and can ensure that effective dehazing can be realized even in the case of limited real fog-free image data.

[0225] Specifically, in the pre-training of the first stage, the Adam optimizer is used empirically, the momentum parameters are set as β1=0.9, β2=0.999, and 100 cycles of training are performed. The initial learning rate is set to 1×10 -4 , and the decay rate is 0.8 every 20 cycles. In the fine-tuning stage, due to the instability of training, the same 100 cycles as the first stage are set. The initial learning rate of the generator and the discriminator is set to 2×10 -4 , and the linear decay strategy is used to adjust every 30 epochs. The weighting values in the generator loss function are set to λ1=1, λ2=0.5, λ3=0.02 and λ4=1 according to the trial and error method.

[0226] As shown in Table 1, Table 1 shows the various evaluation indicators of the application on the SOTS dataset, the URHI dataset and the RTTS dataset.

[0227] Table 1

[0228]

[0229] In the experiment of the synthetic data set SOTS, the performance of the image defogging method based on the generative adversarial network proposed in the application is comprehensively evaluated from both qualitative and quantitative dimensions.

[0230] In the qualitative evaluation link, the application can effectively eliminate the haze residues in the image, does not produce artifacts in the sky area, maintains balanced overall brightness of the image, presents real and natural colors, and at the same time, can significantly restore the texture details obscured by the haze, and greatly improves the visual clarity and realism of the image.

[0231] In the quantitative evaluation aspect, the signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are introduced as two standard indexes widely used in the field of synthetic data defogging research. The experimental results show that on the SOTS data set, the application achieves an average peak signal-to-noise ratio (PSNR) of 34.13 dB and an average structural similarity (SSIM) of 0.9863, both of which reach the current optimal level. In addition, the standard deviation of the model on these indexes is low, which indicates that the performance of the model on different samples has high stability, fully proving the robustness of the model.

[0232] The experiments on the real haze data sets URHI and RTTS also verify the effectiveness of the application in actual scenes. In the experimental process, 300 images are randomly selected from URHI and RTTS for testing. The results show that the application can remove most of the haze residues in real haze images and significantly improve the texture detail performance of the images. Especially in the sky area, it is almost impossible to detect the existence of artifacts. The overall color performance of the image is natural, the brightness distribution is balanced, and the visual effect is obviously better than other methods. In the RTTS data set, the defogging effect of the image is also clear and outstanding. Even in some areas with high-density haze, the application can still effectively process while maximizing the preservation of the original detail information of the image.

[0233] In the quantitative analysis stage, the frequency domain-based analysis method is used as the main evaluation index to measure the defogging quality of the image. The experimental results show that on the URHI data set, the score of the application reaches 27.8128; on the RTTS data set, the score reaches 27.7937, both of which are at a high level. These results fully show that the application can excellently remove the haze in actual complex environments while taking into account the details and realism of the image.

[0234] In addition, two key indicators, edge strength and frequency domain definition, are introduced to further verify the dehazing effect, where the edge strength is calculated by the edge detection algorithm of the Canny operator, and the frequency domain definition is obtained based on the definition evaluation function of the frequency domain analysis. These two indicators can effectively reflect the edge detail retention degree and overall definition of the dehazed image.

[0235] In order to fully demonstrate the effect of the image dehazing method based on the generative adversarial network proposed in the present application in practical application, in this embodiment two, dehazing is taken as a key preprocessing step of target detection task, and its great potential in improving the performance of downstream tasks is verified.

[0236] Figure 5 For comparison of dehazing effect by generative adversarial network, Figure 5 (a) in the above is a foggy image, Figure 5 (b) in the above is a dehazed image, Figure 5 (c) in the above is a label image.

[0237] In this embodiment two, 300 challenging foggy images are randomly extracted from the RTTS dataset, and the target detection model YOLOv8 is used to evaluate the detection performance of the original foggy images and the images processed by different dehazing algorithms respectively.

[0238] By calculating the average precision (AP) of each processing method, this embodiment two reveals the actual impact of different dehazing techniques on the target detection task, and the experimental data fully embodies the excellent performance of the present method. Compared with the untreated foggy image, the image dehazed by the image dehazing method based on the generative adversarial network proposed in the present application improves the average precision of target detection by 4.6%, which is significantly ahead of other dehazing algorithms. In addition, the present application can effectively solve the problem of detection blind area caused by fog and haze, so that many originally undetectable targets can be successfully recognized after dehazing, and at the same time, it can also correct the false classification and positioning caused by fog and haze interference, providing higher accuracy and reliability for the target detection task.

[0239] More notably, the image dehazing method based on the generative adversarial network proposed in the present application not only improves the detection accuracy, but also greatly optimizes the recall rate and comprehensive performance of the model.

[0240] Experiments show that the detection accuracy and recall rate are improved by 4.5% and 3.3% respectively, indicating that the present application can significantly enhance the visibility and information integrity of the image, and provide clearer and more detailed input data for the target detection task.

[0241] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0242] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0243] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0245] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, those skilled in the art can make other variations and modifications of the present application without departing from the present application. Neither requiring nor intending to exhaust all embodiments of the present application, the metes and bounds of the present application are set forth in the claims.

Claims

1. A method for image defogging based on a generative adversarial network, characterized in that, The method comprises the following steps: An image data set is acquired, the image data set comprising different synthetic fog data and real fog data, the synthetic fog data comprising a fog-free image and a corresponding synthetic fog paired image, and the real fog data comprising a real fog image and a corresponding real fog-free image, the preprocessed image data set is divided into a training set and a verification set; The synthetic fog paired image is input into a generator of a generative adversarial network to obtain a dehazed image of the synthetic fog paired image; The dehazed image of the synthetic fog paired image is reconstructed to obtain a reconstructed fog image of the synthetic fog paired image; A sum of a smooth L1 loss between the dehazed image of the synthetic fog paired image and the fog-free image and a smooth L1 loss between the reconstructed fog image and the synthetic fog paired image is taken as a total loss function of the first stage; The generator is trained by the total loss function of the first stage to obtain a first-stage trained generator; The real fog image is input into the generator to obtain a feature map of the real fog image and a dehazed image of the real fog image; The real fog-free image is input into the generator to obtain a dehazed image of the real fog-free image; The dehazed image of the real fog image is input into a post-feature extractor to obtain a dehazed image feature of the real fog image; The feature map of the real fog image is input into a two-layer perception machine network to obtain a multi-layer feature of the real fog image; The dehazed image feature of the real fog image is input into a two-layer perception machine network to obtain a multi-layer dehazed image feature of the real fog image; A multi-layer contrast loss is constructed by calculating a contrast loss function of each spatial position in each feature layer of the real fog image and a dehazed image feature layer of the same layer; A style transfer loss is constructed based on a first dehazed image feature of the real fog image generated by the first-stage trained generator and a second dehazed image feature of the real fog image generated by the second-stage updated generator; An identity consistency loss is constructed based on an L1 norm loss between the real fog-free image and the dehazed image of the real fog-free image; The multi-layer contrast loss, the style transfer loss, the identity consistency loss, a generator loss and a discriminator loss are weighted and summed to obtain a total loss function of the second stage; The generative adversarial network is trained by the total loss function of the second stage, and the trained generative adversarial network is verified by the verification set to obtain a target generative adversarial network; The image to be dehazed is input into the target generative adversarial network to output a dehazed image of the image to be dehazed.

2. The image defogging method based on a generative adversarial network according to claim 1, characterized in that, The dehazed image of the real fog image is input into the post-feature extractor to obtain the dehazed image feature of the real fog image, comprising: After the dehazed image of the real fog image is input into a 3*3 convolution layer, the dehazed image of the real fog image is input into a two-layer 2*2 downsampling layer to obtain a low-resolution feature map of the real fog image; After the low-resolution feature map of the real fog image is input into three residual modules connected in sequence, a dehazed image feature of the real fog image is obtained.

3. The image defogging method based on a generative adversarial network according to claim 1, characterized in that, The formula of the total loss function of the second stage is: ， wherein, is a total loss function for the second stage, is a weight for the multi-layer contrast loss, is the multi-layer contrast loss, is a weight for the style transfer loss, is the style transfer loss, is a weight for the identity consistency loss, is the identity consistency loss, is a weight for the generator loss and the discriminator loss, is the generator loss, is the discriminator loss; Multi-layer contrastive loss The formula is: ， in, Represents a true fog image Follows distribution Find the expected value under the given conditions. The distribution of real fog images in the sample space. To set the number of floors, To select the layer index, For spatial location index, For the last spatial location on each floor, To compare loss functions, For the first real fog image The spatial location in the feature layer of the dehazed image is The feature vector of the sub-block, For the first real fog image The spatial location in each feature layer is The feature vector of the sub-block, In the feature layer of the real fog image, excluding the spatial location... The set of feature vectors of the sub-blocks; Style transfer loss The formula is: ， wherein, a second dehazed image feature of a real hazy image generated by the generator after updating the parameters in the second stage, a first dehazed image feature of a real hazy image generated by the generator trained in the first stage, ; Generator loss The formula is: ， wherein, denotes a dehazed image of a real fog image obeys a distribution under the condition that a mathematical expectation is calculated, a distribution of the dehazed images generated by the generator within the sample space, a discriminator of the generative adversarial network, a dehazed image of a real fog image; discriminator loss The formula is: ， wherein, represents a real fog-free image subject to a distribution under which the mathematical expectation is calculated, is the distribution of real fog-free images within the sample space; Identity consistency loss formula The formula is: ， wherein, denotes the real haze-free image subject to the distribution under which the mathematical expectation is taken, is a dehazed image of the real haze-free image, is a real haze-free image, is the L1 norm.

4. The image defogging method based on a generative adversarial network according to claim 1, characterized in that, The generator of the generative adversarial network comprises: a front feature extractor, a defogging network; wherein the defogging network comprises: a plurality of defogging modules connected in sequence, each defogging module comprising: an image enhancement unit and a defogging unit connected in sequence, and taking the defogging feature map output by the last defogging module as a defogging image; The input feature of the defogging module is input into the defogging module to output a defogging feature map, comprising: The input feature of the defogging module is input into the image enhancement unit to perform feature enhancement to obtain an output enhanced feature map of the image feature enhancement unit; The output enhanced feature map of the image feature enhancement unit is input into the defogging unit to output a defogging feature map, comprising: The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch to output an atmospheric light matrix; The output enhanced feature map of the image feature enhancement unit is input into a transmission matrix estimation branch to output a transmission graph matrix; The atmospheric light matrix, the transmission graph matrix, the output enhanced feature map of the image feature enhancement unit, and the input feature of the defogging module are used to calculate a defogging feature map.

5. The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The input feature of the defogging module is input into the image enhancement unit to perform feature enhancement to obtain an output enhanced feature map of the image feature enhancement unit, comprising: The input feature of the defogging module is dimensionally reduced to obtain a dimensionally reduced input feature of the defogging module; The dimensionally reduced input feature of the defogging module is respectively input into three linear layers to obtain a key vector, a query vector, and a value vector; An attention weight is calculated based on the key vector, the query vector, and the value vector; The formula for calculating the attention weight is: ， wherein, is the output at the th pixel position, is the query vector, is the key vector, is the value vector, is a natural constant, is the transpose, is the square root of the key vector channel dimension, is the index of the value vector, is the index of the key vector, is , is the height of the input feature of the defogging module, is the width of the input feature of the defogging module; The attention weight is dimensionally converted and then up-sampled to obtain a dimensionally converted and up-sampled attention weight; The dimensionally converted and up-sampled attention weight is multiplied by a learnable parameter and then added to the input feature of the defogging module to obtain an output enhanced feature map of the image feature enhancement unit; The formula for obtaining the output enhanced feature map of the image feature enhancement unit by multiplying the dimensionally converted and up-sampled attention weight by the learnable parameter and then adding the input feature of the defogging module is: ， wherein, is an output enhanced feature map of the image feature enhancement unit, is a learnable parameter, is an attention weight after dimension conversion and up-sampling, is an input feature of the defogging module.

6. The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The output enhanced feature map of the image feature enhancement unit is input into an atmospheric scattering matrix estimation branch to output an atmospheric light matrix, comprising: The output enhanced feature map of the image feature enhancement unit is input into a mixed average pooling to obtain a pooled enhanced feature map; The pooled enhanced feature map is sequentially input into a 3x3 convolution layer and a ReLU activation function, and then concatenated with the pooled enhanced feature map to obtain an initial fusion feature; The initial fusion feature is sequentially input into a 3x3 convolution layer and a ReLU activation function, and then concatenated with the initial fusion feature to obtain a secondary fusion feature; The secondary fusion feature is sequentially input into a 1x1 convolution layer and a Sigmoid activation function, and then element-wise added to the initial fusion feature to obtain an atmospheric light pre-generated feature; The atmospheric light pre-generated feature is up-sampled to obtain an atmospheric light matrix.

7. The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The output enhanced feature map of the image feature enhancement unit is input into a transmission matrix estimation branch to output a transmission graph matrix, comprising: The output enhanced feature map of the image feature enhancement unit is processed by a 3*3 convolution layer to obtain a feature map processed by the 3*3 convolution layer; The feature map processed by the 3*3 convolution layer is sequentially processed by a 3*3 convolution layer and a ReLU activation function, and then spliced with the feature map processed by the 3*3 convolution layer to obtain a first spliced feature; The first spliced feature is sequentially processed by a 3*3 convolution layer and a ReLU activation function, and then spliced with the first spliced feature to obtain a second spliced feature; The second spliced feature is sequentially processed by a 3*3 convolution layer and a ReLU activation function, and then spliced with the second spliced feature to obtain a third spliced feature; The third spliced feature is sequentially processed by a 1*1 convolution layer and a Sigmoid activation function, and then element-wise added to the feature map processed by the 3*3 convolution layer to obtain a transmission map matrix.

8. The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The input features of the atmospheric light matrix, the transmission map matrix, the output enhanced feature map of the image feature enhancement unit and the dehazing module are used to calculate a dehazing feature map, and the calculation formula is: ， wherein, is a dehazing feature map, is an output enhanced feature map of the image feature enhancement unit, is an element-wise multiplication, is a transmission map matrix, is an atmospheric light matrix, is an input feature of the dehazing module. 9.The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The dehazing image of the synthetic fog pair image, the atmospheric light matrix output by the atmospheric scattering matrix estimation branch in the dehazing unit of the last dehazing module, and the transmission map matrix output by the transmission matrix estimation branch in the dehazing unit of the last dehazing module are input into an image reconstruction module to obtain a reconstructed fog image, and the calculation formula of the reconstructed fog image is: ， wherein, is a dehazed image of the foggy image, is a dehazed image of the foggy image, is an element-wise multiplication, is a transmission map matrix outputted by a transmission matrix estimation branch in a dehazing unit of the last dehazing module, is an atmospheric light matrix outputted by an atmospheric scattering matrix estimation branch in a dehazing unit of the last dehazing module, is an inverse matrix of is an inverse matrix of 10. The image defogging method based on a generative adversarial network according to claim 4, characterized in that, The image to be dehazed is input into the target generative adversarial network to output a dehazing image of the image to be dehazed, including: The image to be dehazed is input into the front feature extractor to extract a feature map of the image to be dehazed, including: After the image to be dehazed is processed by a 3*3 convolution layer, it is processed by two times of 2 times down-sampling to obtain a low-resolution feature map of the image to be dehazed; The low-resolution feature map of the image to be dehazed is processed by the three residual modules connected in sequence, and then processed by two times of 2 times up-sampling to obtain a feature map of the image to be dehazed; The feature map of the image to be dehazed is input into the dehazing network to output a dehazing image of the image to be dehazed.

Images