A single image dehazing method using detail transfer channel and sub-attention module

Abstract

This invention provides a single-image dehazing method using a detail transfer channel and a sub-attention module. The method includes preparing a training set and a test set of foggy and fog-free images, and further comprising the following steps: training a DITCSA-Net network using images from the training set to generate a training model; saving the trained model to a local folder, testing the trained model using images from the test set, and if satisfactory, saving the trained model as a satisfactory training model; saving the satisfactory training model to a local folder, and testing a blind image of a foggy image using the satisfactory training model. This invention incorporates an attention mechanism into the network encoder module, employing a feature recovery module at the junction of the encoder and decoder to compensate for lost feature information. A sub-attention module is proposed at the decoding layer to prevent information loss during decoding and restore a high-quality fog-free image based on the valid information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a single image defogging method using a detail transfer channel and a sub-attention module. Background Art

[0002] Image dehazing is a fundamental, low-level vision task that has attracted extensive research in recent years. Problems such as low image visibility, blur, and loss of key information adversely impact intelligent applications like autonomous driving and object recognition. These problems are caused by environmental factors such as haze, smog, and fog. Therefore, to mitigate the losses caused by image degradation in adverse weather conditions, recovering haze-free images from degraded images has become a research hotspot.

[0003] Early work on image dehazing primarily focused on estimating the hyperparameters of proposed imaging models for images under hazy skies to restore haze-free images. Due to the ill-posed problem of inaccurate parameter estimation, early prior-based image dehazing methods primarily focused on estimating the medium transmission using the statistical properties of clear images. While these methods achieved effective dehazing results, they were less robust in practice, resulting in inaccurate transmission estimates in most cases. Furthermore, due to parameter inaccuracies, the quality of images recovered from physical models was severely degraded.

[0004] Convolutional neural network (CNN) methods were later proposed to accurately estimate transmission, thereby addressing inaccuracies in transmission estimation. These data-driven methods significantly improve the accuracy of estimated transmission, thus outperforming prior-based methods. However, these CNN methods are limited to valid scenes with physical models. As a result, they are unable to recover high-quality haze-free images in most cases, especially those that do not fit the model scene, resulting in poorly restored images.

[0005] To address this issue, end-to-end image dehazing methods have been developed that directly train the mapping between hazy and haze-free images using deep learning networks. These methods eliminate the need to estimate transmission map parameters, atmospheric light, and recover the haze-free image using physical models, resulting in excellent image recovery performance. These methods have attracted significant research attention in feature processing, but they neglect the importance of detail information in image dehazing. Although the haze-free images recovered using most dehazing methods contain information about ground-truth feature regions, the loss of detailed information leads to image quality degradation, such as reduced contrast and blurring.

[0006] In the June 2020 issue of Baidu Academic, Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang published a paper titled "Multi-Scale Boosted Dehazing Network with Dense Feature Fusion." In this paper, they proposed a multi-scale boosted dehazing method based on a U-Net dense feature fusion network architecture. The proposed method is based on two principles, boosting and error feedback, and demonstrates their applicability to the dehazing problem. By incorporating a boosting-operation-subtraction boosting strategy into the decoder of the proposed model, a simple yet effective boosted decoder is developed to progressively restore the haze-free image. This addresses the spatial preservation issue. Within the U-Net information architecture, a dense feature fusion module employing a back-projection feedback scheme is designed. Experimental results demonstrate that the dense feature fusion module can simultaneously compensate for missing spatial information in high-resolution features and utilize non-adjacent features. Extensive evaluation demonstrates that the proposed model performs well compared to state-of-the-art methods on benchmark datasets and real-world hazy images. The disadvantages of this method are the lack of processing of detail information and the low model migration performance, and it cannot solve the defogging problem of real-world foggy images well. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a single image dehazing method using a detail transmission channel and a sub-attention module. The attention mechanism is introduced into the network encoder module, and a feature recovery module is used at the connection between the encoder and decoder to compensate for the lost feature information. A sub-attention module is proposed in the decoding layer to avoid information loss during the decoding process and restore a high-quality haze-free image based on the effective information.

[0008] The present invention provides a single image defogging method using a detail transfer channel and a sub-attention module, comprising preparing a training set and a test set of foggy and fog-free image sets, and further comprising the following steps:

[0009] Step 1: Use the images in the training set to perform DITCSA-Net network training to generate a training model; Step 2: Save the training model to a local folder, use the images in the test set to test the effect of the training model, and if satisfactory, save the training model as a satisfactory training model;

[0010] Step 3: Save the satisfactory training model to a local folder, and use the satisfactory training model to test the blind image of the foggy image.

[0011] Preferably, the DITCSA-Net network includes an encoder module, a multi-scale feature enhancement and recovery module, a detail information transmission channel and a decoder module.

[0012] In any of the above solutions, preferably, the encoder module is used to extract a feature map of the blurred image and promote the feature map to a high-dimensional space to process feature information.

[0013] In any of the above solutions, preferably, the encoder module is embedded with a feature attention submodule, and a dense residual group is added after each layer.

[0014] In any of the above solutions, preferably, a maximum pooling layer with a 5×5 convolution kernel is added before the average pooling layer of the feature attention submodule.

[0015] In any of the above solutions, preferably, the multi-scale feature enhancement and restoration module is used to improve the resolution of the feature map and restore the information lost during the processing of the feature map.

[0016] In any of the above solutions, preferably, the detail information transmission channel is a detail information transmission channel using a jump connection technology, which is expressed as

[0017]

[0018] in, A tributary information representing the details of the information, ρ is the parameterized positive linear unit, δ is batch normalization (BN), Represents the input information of the encoder layer, I D Represents detailed information sent to the decoding layer.

[0019] In any of the above solutions, preferably, a high-dimensional feature map containing rich information is generated through processing of the encoder module, the multi-scale feature enhancement and recovery module, and the detail information transmission channel.

[0020] In any of the above solutions, preferably, the decoder module is used to restore the haze-free image, reduce the distortion rate and improve the contrast of the haze-free image.

[0021] In any of the above solutions, preferably, the decoder module is further used to restore detail information of the haze-free image and the low-dimensional haze-free image according to the high-dimensional feature map.

[0022] In any of the above schemes, preferably, the decoder module includes a sub-attention sub-module for implementing an attention-based link between the encoder and the decoder.

[0023] In any of the above schemes, preferably, the sub-attention is expressed as follows:

[0024] F out =Conv(F in )*σ(Conv(Conv(F in )))+F in

[0025] Among them, F in Represents the local restored feature map of the input, F out is the output of the sub-attention module.

[0026] In any of the above solutions, preferably, the input local restoration feature map F in Multiply the output of a simple 3×3 convolution layer with two layers of 3×3 convolution and a Sigmoid activation function, plus the input local recovery feature map F in Get the output F of the sub-attention module out .

[0027] This paper proposes a single image defogging method using a detail transfer channel and a sub-attention module. The concept is simple, and it can effectively preserve detailed information and restore high-quality haze-free images.

[0028] DITCSA-Net refers to Single Image Dehazing Network with Detailed Information Transmission Channel and Sub Attention, that is, a single image dehazing network with detailed information transmission channel and sub-attention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart of a preferred embodiment of a single image defogging method using a detail transfer channel and a sub-attention module according to the present invention.

[0030] Figure 2 Schematic diagram of the network structure of an embodiment of the DITCSA-Net network of the single image dehazing method using the detail transmission channel and the sub-attention module according to the present invention.

[0031] Figure 3 Schematic diagram of the coding layer structure of an embodiment of the DITCSA-Net network of the single image dehazing method using the detail transmission channel and the sub-attention module according to the present invention.

[0032] Figure 4 Schematic diagram of the structure of a multi-scale feature enhancement and restoration module of an embodiment of the DITCSA-Net network of the single image dehazing method using the detail transmission channel and the sub-attention module according to the present invention.

[0033] Figure 5 Schematic diagram of the detail information transmission channel structure of an embodiment of the DITCSA-Net network of the single image dehazing method using the detail transmission channel and the sub-attention module according to the present invention.

[0034] Figure 6 Schematic diagram of the sub-attention module structure of an embodiment of the DITCSA-Net network of the single image dehazing method using the detail transmission channel and the sub-attention module according to the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] like Figure 1 As shown, step 100 is performed to prepare a training set and a test set of foggy and non-foggy image sets.

[0038] Execute step 110 to perform DITCSA-Net network training using the images in the training set to generate a training model.

[0039] The DITCSA-Net network includes an encoder module, a multi-scale feature enhancement and recovery module, a detail information transmission channel and a decoder module.

[0040] The encoder module extracts feature maps from blurred images and elevates them to a high-dimensional space to process feature information. It embeds a feature attention submodule and adds a dense residual group after each layer. A max pooling layer with a 5×5 convolution kernel is added before the average pooling layer in the feature attention submodule.

[0041] The multi-scale feature enhancement and restoration module is used to improve the resolution of the feature map and restore the information lost during the feature map processing.

[0042] The detail information transmission channel is a detail information transmission channel using the jump connection technology, which is expressed as

[0043]

[0044] in, A tributary information representing the details of the information, ρ is the parameterized positive linear unit, δ is batch normalization (BN), Represents the input information of the encoder layer, I D Represents detailed information sent to the decoding layer.

[0045] Through the processing of the encoder module, the multi-scale feature enhancement and recovery module and the detail information transmission channel, a high-dimensional feature map containing rich information is generated.

[0046] The decoder module is used to restore the haze-free image, reduce the distortion rate, and improve the contrast of the haze-free image. It restores the detail information of the haze-free image and the low-dimensional haze-free image based on the high-dimensional feature map. The decoder module includes a sub-attention submodule for implementing an attention-based link between the encoder and decoder. The sub-attention is expressed as follows:

[0047] F out =Conv(F in )*σ(Conv(Conv(F in )))+F in

[0048] Among them, F in Represents the local restored feature map of the input, F out is the output of the sub-attention module.

[0049] Input local restored feature map F in Multiply the output of a simple 3×3 convolution layer with two layers of 3×3 convolution and a Sigmoid activation function, plus the input local recovery feature map F in Get the output F of the sub-attention module out .

[0050] Execute step 120 to save the training model to a local folder, use the images in the test set to test the effect of the training model, and if satisfactory, save the training model as a satisfactory training model.

[0051] Execute step 130 to save the satisfactory training model to a local folder, and use the satisfactory training model to test the blind image of the foggy image.

[0052] Example 2

[0053] The present invention is a deep learning image dehazing algorithm based on convolutional neural networks. A trainable end-to-end dehazing network named "Single Image Dehazing Network with Detailed Information Transmission Channel and Sub Attention" (DITCSA-Net) based on an encoder-decoder architecture and a U-Net backbone network is proposed.

[0054] The contents of DITCSA-Net include: an encoder module based on the attention mechanism, a detail information transmission channel, a multi-scale feature enhancement and recovery module, and a sub-attention module in the decoder.

[0055] DITCSA-Net training requires a dataset of labeled "foggy and clean" images to generate a training model. The network hyperparameters from the trained model are then incorporated into the network model to achieve dehazing of foggy images. It effectively preserves detailed information and restores high-quality clean images. Hazy images contain both true information corresponding to the ground truth and invalid information generated by fog. An attention mechanism is introduced into the proposed network encoder module to accurately extract valid feature information from hazy images, rather than invalid information from the foggy portion.

[0056] Steps to use DITCSA-Net (implemented in Python programming language):

[0057] 1. Prepare a training set of "fog-no-fog" images and set the image format to ".jpg" or ".png";

[0058] 2. Use the training file "trian.py" to start network training. Adjust the network parameters such as batch size and lr as needed. The default settings are "batchSize = 1, lr = 1e-4, epochs = 1000";

[0059] 3. Save the trained model to a local folder and use the "vali.py" file and the test set to test the effectiveness of the network training model. If you are not satisfied, you can use the "checkpoint" technology to continue training a satisfactory model.

[0060] 4. Save the satisfactory trained model to a local folder and use "test.py" to test the blind image of the foggy image.

[0061] Example 3

[0062] To reduce the loss of image information in end-to-end tasks and improve image restoration, we reconsidered the image dehazing task. Inspired by the successful application of the encoder-decoder architecture in many image processing tasks, such as face detection, image super-resolution, and image denoising, we carefully designed the encoder and decoder structures using this architecture. Inspired by the U-Net and attention mechanism, we proposed a dehazing network, called DITCSA-Net, that utilizes detail information transmission channels and sub-attention.

[0063] like Figure 2As shown in the figure, the overall network architecture inherits the basic structure of U-Net and densely connected networks. However, unlike these architectures, 18 multi-scale feature enhancement and recovery modules are embedded in the connection between the encoder and decoder modules. Four dense residual groups are also added after each encoder layer, and sub-attention modules are used in each decoder. Finally, four detail information transmission channel blocks are added between the encoder and decoder at the same layer. DITCSA-Net consists of an encoder module, a feature recovery module, a decoder module, and a detail information transmission channel module. The encoder module is mainly used to extract the feature map of the blurred image and elevate it to a high-dimensional space to process the feature information. The feature recovery module is used to increase the resolution of the feature map and recover the information lost during the feature map processing. The decoder module is the core module for restoring the haze-free image, reducing the distortion rate, and improving the contrast of the haze-free image.

[0064] The work of each module is as follows:

[0065] 1. Encoder module (such as Figure 3 shown)

[0066] Multi-layer encoders perform very well in image feature extraction. However, in image processing tasks, this is not the optimal approach for extracting essential information from images with important features. The attention mechanism has become a mainstream component in deep neural networks because its mechanism of ignoring irrelevant information and focusing on important information is effectively expressed in deep learning networks. A feature attention module is embedded in the encoder, and a dense residual group is added after each layer. As considered in the attention mechanism, different channels have different importance. By using learned weights to assign corresponding scales to different channels, the original features are rescaled. During feature processing, weighted channels can focus on more important features based on their corresponding weights, improving network performance. Specifically, the channel attention module is improved. A max pooling layer with a 5×5 convolution kernel is introduced before the average pooling layer. Existing algorithms convert an H×W×C image to a 1×1×C image by squeezing it along the dimensions, and then learn a weight W through an excitation operation. Although images are composed of pixels, analyzing an image from a single pixel loses its structural integrity and destroys its spatial structure. To avoid this problem, a 5×5 max pooling layer is added, which increases the receptive field and splits the image to reduce spatial corruption.

[0067] 2. Multi-scale feature enhancement and restoration module (such as Figure 4 shown)

[0068] Although the attention mechanism encoder achieves effective expression in extracting local feature information, the loss of spatial information is an inevitable challenge to the destruction of image structure. Images with structural damage have low contrast, poor visibility, and severe distortion, which affects the quality of the restored image. A multi-scale feature enhancement and recovery block is proposed. It is known that the depth of the neural network is of great significance to its performance. The increase in the depth of deep learning neural networks mainly leads to more effective representation. When optimizing weight parameters, training deeper networks brings more significant challenges, which can easily produce vanishing gradients and network degradation. The performance of ResNet is used to avoid this problem. Improving the feature information of deep high-dimensional feature maps can enhance the weight of features and compensate for the lost information in shallow layers to restore the integrity of the image.

[0069] 3. Detailed information transmission channel (such as Figure 5 shown)

[0070] The encoder focuses on more important feature information, while the detail information that is ignored during image restoration depends entirely on the generation of the retained feature information. The image restored using this method will result in severe distortion, especially in non-uniform haze. For the image dehazing task, ignoring the fog information and retaining the detail information of the haze-free image and transmitting it to the decoding layer for image restoration is a huge challenge. To this end, a detail information transmission channel using skip connection technology is proposed. It is expressed as:

[0071]

[0072] Here, ρ is the parameterized rectified linear unit (PReLU) and δ is batch normalization (BN). Represents the input information of the encoder layer, I DRepresents the detail information sent to the decoder layer. The detail information transfer channel is a non-local global information operation that captures the dependency on local feature information. Adding two convolutional layers helps compress local feature information, while skip connections are used to diffuse global information, allowing for more detailed information to be used in the decoder layer. Batch normalization (BN) and convolutional layers are combined to ensure that the complete feature map of the decoder layer is recovered from the focused feature information of the encoding layer. DITC only compensates for detail information to prevent feature map distortion and poor recovery. It normalizes the transmitted information and extracts features to obtain detailed information, which is relatively effective in restoring the true image and ignoring redundant information. Redundant information can lead to image quality degradation such as excessive brightness, loss of texture detail, and local blurring. Each DITC is applied to the corresponding encoding and decoding layer to match the corresponding feature map size. Parallel connections are used to enrich the detail information of the encoding and decoding layers, thereby improving the quality of the dehazed image. The detail transfer channel is a key method for preserving detailed information that is easily lost during the encoding process. It is used between the encoder and decoder layers at the same level where feature information is transmitted to reduce the loss of detailed information during encoding.

[0073] 4. Through the processing of the previous modules of DITCSA-Net, an information-rich high-dimensional feature map is obtained. The main task of the decoder is to restore the detail information of the haze-free image and the low-dimensional haze-free image based on the feature map. In the traditional encoder-decoder architecture, the decoder can adopt a neural network or other models as an independent model. However, since the decoder cannot utilize enough effective information, the low connectivity with the feature map that the encoder focuses on leads to poor image restoration. The attention mechanism is introduced into the decoder module, and an attention-based link is implemented between the encoder and decoder. It allows the decoder to utilize effective feature information to improve the performance of the network. The attention mechanism is incorporated into the decoder, and a sub-attention module (such as Figure 6 Sub-attention is introduced in each decoding layer to enhance the decoder's performance in recovering low-dimensional feature maps. Connections are established between non-adjacent layers to improve the utilization of effective feature information and enhance image dehazing performance. The sub-attention is expressed as follows:

[0074] F out =Conv(F in )*σ(Conv(Conv(F in )))+F in

[0075] Among them, F in Represents the local restored feature map of the input, F out Is the output of the sub-attention module. The local restored feature map F of the input inFirst, multiply the output of a simple 3×3 convolution layer with two layers of 3×3 convolution and a Sigmoid activation function, and finally add the input local recovery feature map F in The output of Sub Attention is obtained. Each layer of decoder recovers more effective information and generates an attention feature map in the recovered state, which is effectively utilized in the next layer of decoder, ultimately achieving the image dehazing effect.

[0076] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A single image defogging method using a detail transfer channel and a sub-attention module, comprising preparing a training set and a test set of foggy and fog-free image sets, characterized in that: The following steps are also included: Step 1: Use the images in the training set to perform DITCSA-Net network training to generate a training model. The DITCSA-Net network includes an encoder module, a multi-scale feature enhancement and recovery module, a detail information transmission channel, and a decoder module; The encoder module is embedded with a feature attention submodule and a dense residual group is added after each layer; Add a maximum pooling layer with a 5×5 convolution kernel before the average pooling layer of the feature attention submodule; The detailed information transmission channel is a detailed information transmission channel using a jump connection technology, which is expressed as in, A tributary information representing the details of the information, ρ is the parameterized positive linear unit, δ is batch normalization (BN), Represents the input information of the encoder layer, I D Represents the detailed information sent to the decoder layer; Generate a high-dimensional feature map containing rich information through processing by the encoder module, the multi-scale feature enhancement and recovery module, and the detail information transmission channel; The decoder module is used to restore the detail information of the haze-free image and the low-dimensional haze-free image according to the high-dimensional feature map. The decoder module includes a sub-attention submodule for implementing an attention-based link between the encoder and the decoder. The sub-attention is expressed as follows: F out =Conv(F in )*σ(Conv(Conv(F in )))+F in Among them, F in Represents the local restored feature map of the input, F out is the output of the sub-attention module; The local restored feature map F of the input in Multiply the output of a simple 3×3 convolution layer with two layers of 3×3 convolution and a Sigmoid activation function, plus the input local recovery feature map F in Get the output F of the sub-attention module out ; Step 2: Save the training model to a local folder, use the images in the test set to test the effect of the training model, and if satisfactory, save the training model as a satisfactory training model; Step 3: Save the satisfactory training model to a local folder, and use the satisfactory training model to test the blind image of the foggy image.

Images