An image dehazing method based on improved GAN

Abstract

This invention discloses an image dehazing method based on an improved GAN, belonging to the field of image dehazing technology. The method includes the following steps: acquiring a hazy image; inputting the hazy image into a generator network model based on an improved GAN, and outputting a dehazed image. This invention uses a generative adversarial network as its basic framework, incorporating a residual module that fuses attention and a dehazing module that connects to the context into the generator, and using an improved PatchGAN discriminator network structure to supervise the image generated by the generator. This results in a dehazed image that is clearer overall, easier to observe, and has richer colors that more closely resemble the real image.

Description

Technical Field

[0001] This invention relates to the field of image dehazing technology, and in particular to an image dehazing method based on an improved GAN. Background Technology

[0002] Image dehazing is a crucial research area in computer vision. It shows significant advancements in model ensemble, global optimization, real-time processing, and multimodal image dehazing. Furthermore, image dehazing technology has been widely applied in numerous fields. For example, in monitoring systems for large infrastructure such as airports, ports, and bridges, removing heavy fog ensures clear visibility, improving safety and security. In the field of autonomous driving, existing vehicle cameras struggle to capture sufficient information in low-visibility conditions; image dehazing helps them capture this information. In military reconnaissance and security, removing fog interference allows for better surveillance and investigation. In photography, where images are often imperfect, dehazing can help salvage some situations and enhance the aesthetics of the photograph. With continuous improvements in computer hardware performance and algorithm theory, image dehazing technology is rapidly evolving.

[0003] Early image dehazing techniques were primarily based on traditional mathematical models, such as Retinex and Dark Channel Prior. However, these traditional methods have inherent limitations, such as their inability to handle complex haze scenes and their inability to improve the attenuation process. In recent years, with the development of deep learning technology, image dehazing algorithms based on convolutional neural networks (CNNs) have gradually gained widespread application. Some of the earliest deep learning models include DehazeNet, MSCNN, AOD-Net, and DCPDN. DehazeNet is a deep framework that uses neural networks combined with normalized convolutions to improve performance on computer vision tasks such as alternating optimization. This model was the first to use deep neural networks for this field, effectively achieving excellent results. MSCNN, proposed in 2016, has the ability to remove multiple foam residues in natural scenes while also supporting dehazing of occluded areas. AOD-Net, based on fully convolutional methods, is simpler than MSCNN and has advantages over traditional algorithms based on image attenuation models. DCPDN is often compared to MSCNN and uses a noisy CNN network to achieve high-resolution images at low resolutions. Furthermore, due to increasingly severe smog pollution, several video-based image dehazing methods have emerged in recent years, such as FastDehazing and DynamicModeDecomposition. These methods utilize multi-frame image information captured in a video, combined with temporal correlation, to maximize visual quality. In summary, these deep learning models not only enhance the restoration capabilities of single images but can also be applied to handling blurring variations in video sequences.

[0004] However, the aforementioned deep learning-based image dehazing algorithms suffer from issues such as missing image feature details and incomplete removal of complex fog. Summary of the Invention

[0005] This invention provides an image dehazing method based on an improved GAN, which solves the problems of incomplete dehazing, loss of feature details, and blurred textures after dehazing in existing dehazing networks.

[0006] This invention provides an image dehazing method based on an improved GAN, comprising the following steps:

[0007] Acquire a foggy image;

[0008] The hazy image is input into a generator network model based on an improved GAN, and the dehazed image is output, including:

[0009] The foggy image is convolved using a convolution module to obtain a low-dimensional feature map.

[0010] An attention map is obtained by performing high-dimensional feature extraction and weighted fusion on the low-dimensional feature map based on multiple residual feature extraction modules.

[0011] The context-based dehazing module performs convolution on the hazy image and the attention map to obtain the dehazed image.

[0012] Preferred options also include:

[0013] Obtain a fog-free image;

[0014] The haze-free image and the haze image are input into the discriminator, and the discriminant result is output.

[0015] The generator network model based on the improved GAN is trained according to the discrimination results, and the trained model is used to dehaze the foggy image.

[0016] Preferably, the convolution module includes a convolutional layer, a normalization layer, and an activation function.

[0017] Preferably, the attention map is obtained by performing high-dimensional feature extraction and weighted fusion on the low-dimensional feature map based on multiple residual feature extraction modules, including the following steps:

[0018] Multiple high-dimensional features are extracted from the low-dimensional feature map by three residual attention modules of different scales, resulting in multiple high-dimensional feature maps with different channels.

[0019] We perform weighted fusion on high-dimensional feature maps from multiple channels, select features that fit the current scale, and replace the original features with these features to obtain an attention map.

[0020] Preferably, the dehazed image is obtained by convolving the hazy image and the attention map using a context-based dehazing module, including the following steps:

[0021] Feature extraction is performed on foggy images and attention maps using 12 convolutional layers;

[0022] The extracted features are input into the Tanh activation function, and the output is a dehazed image.

[0023] Preferably, the extracted features are refined using the ELU activation function after each multi-layer convolution operation.

[0024] Preferably, the improved GAN-based generator network is trained using a perceptual loss function and a conditional loss function, wherein the perceptual loss function is as follows:

[0025]

[0026] In the formula, L content Indicates perceived loss. To represent the difference in feature maps between dehazed and hazy images, W c,kH represents the width of the feature map. c,k The height of the feature map is represented by S, the real haze-free image is represented by G(B), the haze-free image generated by the generator is represented by c, the number of convolutional layers is represented by k, the number of pooling layers is represented by x, the x-coordinate of the current pixel is represented by y, and the y-coordinate of the current pixel is represented by y.

[0027] The conditional loss function is as follows:

[0028]

[0029] In the formula, L condition It is the conditional loss, where A is the score given by the discriminator to the image generated by the generator. p is a variable in a specific image. g D is the image generated by the generator, D is the probability value of the discriminator judging whether the image generated by the generator is real or fake, and B is the image to be dehazed in the input network.

[0030] Preferably, the discriminator is a PatchGAN discriminator.

[0031] Preferably, the discriminator is trained using a discriminative loss function, which is shown below:

[0032]

[0033] In the formula, L adv To determine the loss, p r It is a real and clear image, p g A represents the image generated by the generator, and A is the score given by the discriminator to the image generated by the generator. This indicates gradient calculation. This indicates that sampling is performed uniformly along a straight line from p. r and p g Point pairs sampled in the middle.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention employs a generative adversarial network (GAN) as its basic framework. It incorporates a residual module with fused attention and a dehazing module that considers context into the generator. The residual module focuses on contour and texture information, while the dehazing module acquires the spatial dimensions of the image. Furthermore, an improved PatchGAN discriminator network structure is used to supervise the generator's image generation. This results in dehazed images that are clearer, easier to observe, and have richer colors that more closely resemble realistic images. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of an image dehazing method based on an improved GAN according to the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the improved GAN generator network of the present invention;

[0039] Figure 3 This is a schematic diagram of the residual feature extraction module of the present invention;

[0040] Figure 4 This is a schematic diagram of the context dehazing module of the present invention;

[0041] Figure 5 This is a schematic diagram of the discriminator of the present invention;

[0042] Figure 6 (a) is a schematic diagram of a foggy day in this embodiment;

[0043] Figure 6 (b) is a schematic diagram of the dehazed image obtained by the DCP algorithm;

[0044] Figure 6 (c) is a schematic diagram of the dehazed image obtained by the CAP algorithm;

[0045] Figure 6 (d) is a schematic diagram of the dehazed image obtained by the AODNet algorithm;

[0046] Figure 6 (e) is a schematic diagram of the dehazed image obtained by the EPDN algorithm;

[0047] Figure 6 (f) is a schematic diagram of the dehazed image obtained by the URNet algorithm;

[0048] Figure 6 (g) is a schematic diagram of the dehazed image obtained by the algorithm of the present invention;

[0049] Figure 6 (h) is a schematic diagram of a clear, fog-free image in this embodiment. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Reference Figure 1 To remove noise such as haze from images while restoring more texture details, this invention provides an image dehazing method based on an improved GAN, specifically including the following steps:

[0052] Step 1: Acquire foggy and fog-free images. Data is collected using a camera to obtain foggy and fog-free images.

[0053] Step 2: Refer to Figure 2 The process involves inputting a foggy image into a generator network model based on an improved GAN, and outputting a dehazed image, including:

[0054] (1) Convolve the foggy image based on the convolution module to obtain a low-dimensional feature map.

[0055] A foggy image of length W and width H is taken as input and passed through a convolutional kernel of size 7 to obtain a low-dimensional feature map. An instance normalization layer and an activation function are added after each convolutional layer to reduce the risk of gradient vanishing.

[0056] (2) Based on multiple residual feature extraction modules, high-dimensional feature extraction and weighted fusion are performed on the low-dimensional feature map to obtain the attention map.

[0057] Three residual feature extraction modules of different scales are connected in series to obtain spatial information of different receptive fields, thereby increasing the depth and expressiveness of the network. Each module is based on the residual network and adopts a channel attention mechanism to enhance the attention to key feature channels, while retaining low-level information and passing it to the deep layers.

[0058] Reference Figure 3 The residual module can alleviate the gradient explosion problem caused by deep learning during feature extraction, while the channel attention mechanism can extract rich features and reduce unnecessary computational and storage resource consumption. Combining these two approaches, a residual attention module was built. The network first extracts high-dimensional features of the image using three residual blocks, and then, by weighting the feature maps of different channels, selects salient features that fit the current scale to replace the original features. This allows the main network to focus on effective information, more effectively handle thick and thin fog regions, and improve feature extraction efficiency and network propagation performance.

[0059] (3) The hazy image and the attention map are convolved based on the context dehazing module to obtain the dehazed image.

[0060] Reference Figure 4 The context-based dehazing module, acting as an encoder-decoder network, takes the original image and attention image as input. It employs 12 convolutional layers to extract the spatial context of the image. After each convolutional layer, the ELU activation function refines the extracted features, restoring local image texture details and preventing overfitting during training. Finally, the dehazed image is output after passing through the Tanh activation function. This context-based dehazing module improves color correction and detail feature recovery capabilities, ensuring sharp image edges.

[0061] The generator network model based on the improved GAN is trained using a loss function. The loss function used in this invention is based on WGAN-GP, and the overall loss of the generator includes perceptual loss and conditional loss, as shown in the following equation:

[0062] L G =L condition +λ*L content

[0063] In the formula, L is used condition Conditional loss replaces the original WGAN-GP feature loss, aiming to ensure that the generator's output image does not deviate from the input foggy image in terms of color, brightness, and detail, thus guaranteeing the realism of the generated image. λ is 100.

[0064] The conditional loss function is shown in the following equation:

[0065]

[0066] In the formula, L condition It is the conditional loss, where A is the score given by the discriminator to the image generated by the generator. p is a variable in a specific image. g D is the image generated by the generator, D is the probability value of the discriminator judging whether the image generated by the generator is real or fake, and B is the image to be dehazed in the input network.

[0067] L content The perceptual loss is used to calculate the difference in feature maps between the dehazed image generated by the generator after passing through the conv3-3 layers of the VGG19 network and the real hazy image. It addresses the problem of unclear target information and blurry images caused by the classic content loss L1 when optimizing specific targets, ensuring that the generated image has clear details and sharp contours. The perceptual loss is shown in the following equation:

[0068]

[0069] In the formula, The difference W between the dehazed image generated by the generator and the feature map of the real sharp image after passing through the c-th convolutional layer and the k-th pooling layer of the VGG19 network. c,k H represents the width of the feature map. c,k G(B) represents the height of the feature map, S is the real haze-free image, G(B) is the dehaze image generated by the generator, x is the x-coordinate of the current pixel, and y is the y-coordinate of the current pixel.

[0070] Step 3: Input the hazy image and the hazy image into the discriminator and output the discrimination result.

[0071] Reference Figure 5 The discriminator of this invention adopts the basic network framework of PatchGAN. The output of the PatchGAN discriminator is regarded as a matrix, and the elements in the matrix represent the discrimination results of the corresponding regions in the original image. The advantage of this structure is that it can comprehensively evaluate and judge different regions of the image, rather than judging the entire image.

[0072] The discriminant loss of WGAN-GP is used as the loss function of the discriminator, and the discriminant loss is shown in the following formula:

[0073]

[0074] In the formula, L adv To determine the loss, p r It is a real and clear image, p g A represents the image generated by the generator, and A is the score given by the discriminator to the image generated by the generator. This indicates gradient calculation. This indicates that sampling is performed uniformly along a straight line from p. r and p g Point pairs sampled in the middle.

[0075] In summary, the overall loss function of this invention is shown in the following equation:

[0076] L G =L condition +λ*L content +L adv

[0077] This method uses the basic network framework of PatchGAN for the discriminator. Compared with the traditional global discriminator, the advantage of using a local discriminator is that it comprehensively considers the influence of various parts of the image. The input is a clear image and the generated image, and the output is a K*K matrix. The Kth value corresponds to the score of the Kth small block. By averaging all the scores and feeding them back to the generator, the edge details of the image can be improved.

[0078] Step 4: Train the improved GAN generator network based on the discrimination results, and use the trained improved GAN generator network to dehaze the foggy image.

[0079] Reference Figure 6 The algorithm of this invention can better restore the image and retain more details during the dehazing process. The restored colors are more saturated and closer to the real image, and it can be used for image dehazing in different scenarios.

[0080] As shown in Table 1, a quantitative comparison was made between the current mainstream dehazing algorithms DCP, CAP, AODNet, EPDN, and URNet and the algorithm proposed in this invention based on the SOTS test set. The table shows that the algorithm proposed in this invention has significant improvements in objective indicators such as PSNR and SSIM. The algorithm proposed in this invention is on average 22.75% higher than other algorithms in terms of PSNR and 10.49% higher than other algorithms in terms of SSIM.

[0081] Table 1 Quantitative Evaluation of Defogging Experiment

[0082]

[0083] To address the shortcomings of existing dehazing networks, such as incomplete dehazing, loss of feature details, and blurred textures after dehazing, this invention employs a Generative Adversarial Network (GAN) as its basic framework. A residual module with fused attention is added to the generator to focus on contour and texture information. A context-sensitive dehazing module is used to obtain the spatial dimension of the image. Perceptual loss and conditional loss are used to ensure the realism of the image, and an improved PatchGAN discriminator network structure is used to supervise the generator's image generation. The dehazed image restored by this invention is generally clear, easy to observe, and has rich colors that more closely resemble the real image.

[0084] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0085] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An improved GAN-based image defogging method, characterized in that, The method comprises the following steps: obtaining a foggy image; inputting the foggy image into an improved GAN-based generator network model to output a defogged image, comprising: convolving the foggy image based on a convolution module to obtain a low-dimensional feature map; extracting high-dimensional features from the low-dimensional feature map based on multiple residual feature extraction modules and performing weighted fusion to obtain an attention map; convolving the foggy image and the attention map based on a context defogging module to obtain a defogged image; the convolution module comprises a convolution layer, a normalization layer and an activation function; extracting high-dimensional features from the low-dimensional feature map based on multiple residual feature extraction modules and performing weighted fusion to obtain an attention map, comprising the following steps: extracting multiple high-dimensional features of the low-dimensional feature map through three residual attention modules of different scales to obtain multiple high-dimensional feature maps of different channels; performing weighted fusion on the multiple high-dimensional feature maps of different channels to filter out features conforming to the current scale and replacing the original features with the features to obtain an attention map; convolving the foggy image and the attention map based on a context defogging module to obtain a defogged image, comprising the following steps: extracting features from the foggy image and the attention map through 12 layers of convolution; inputting the extracted features into a Tanh activation function to output a defogged image; after the multi-layer convolution operation, the extracted features are refined through an ELU activation function.

2. The image defogging method based on improved GAN of claim 1, wherein, Further comprising: obtaining a fog-free image; inputting the fog-free image and the foggy image into a discriminator to output a discrimination result; training the improved GAN-based generator network model according to the discrimination result, and defogging the foggy image through the trained model.

3. The improved GAN-based image defogging method of claim 2, wherein, The improved GAN generator network is trained through a perception loss function and a conditional loss function, wherein the perception loss function is as follows: In the formula, represents a perception loss, is a difference between the defogging image and the fog-free image feature map, represents a width of the feature map, represents a height of the feature map, and S is a real fog-free image, is a defogging image generated by the generator, c is a number of convolution layers, k is a number of pooling layers, x is an abscissa of the current pixel point, y is an ordinate of the current pixel point. the conditional loss function is as follows: wherein L condition is a conditional loss, A is a value scored by the discriminator for the image generated by the generator, is a variable in the image generated by the generator, is the image generated by the generator, D is a probability value of the discriminator judging whether the image generated by the generator is true or false, B is an image to be defogged in the input network.

4. The image defogging method based on improved GAN of claim 2, wherein, the discriminator adopts a PatchGAN discriminator.

5. The improved GAN-based image defogging method of claim 2, wherein, The discriminator is trained through a discrimination loss function, and the discrimination loss function is as follows: wherein L adv is the loss to discriminate, is the real clear image, is the image generated by the generator, A is the value scored by the discriminator for the image generated by the generator, denotes the gradient computation, denotes sampling uniformly along a straight line from and the pair of points sampled from

Images