A multi-patch dehazing method based on dual-attention level feature fusion

Abstract

The present invention discloses a multi-patch dehazing method (DAMPHN) based on dual-attention hierarchical feature fusion, comprising the following steps: obtaining a training sample dataset: obtaining an existing clear / hazy dataset with non-uniform haze characteristics, and performing data preprocessing according to training requirements. A linear combination of reconstruction loss, perceptual loss, and total variation loss is selected as the loss function; and the DAMPHN network model is trained. The present invention solves the problem of poor image dehazing effect caused by insufficient hierarchical feature fusion capability in the prior art, while ensuring that the complexity of the algorithm is not increased, by simplifying the encoder-decoder structure and adding a dual-attention hierarchical feature fusion module composed of channel and pixel attention between different levels. The present invention has the best dehazing result, is strong in color and detail restoration, and has the best dehazing effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image fog-free technology, and in particular to a multi-patch defogging method based on dual-attention level feature fusion. Background Art

[0002] Images captured by visible light cameras in foggy conditions suffer from background blur, reduced contrast, and missing feature information, posing significant challenges for subsequent advanced visual information processing. For example, in autonomous driving, foggy conditions can severely impact the camera's field of view, reducing the accuracy of visual environmental perception. Furthermore, the highly random distribution of haze particles in real-world environments can introduce non-uniform noise into images. Mountainous areas, for example, are prone to advection fog and uphill fog, which exhibit extremely non-uniform distribution characteristics and pose challenges for drone-based power inspections. Similarly, image dehazing has long been one of the top ten research areas at the Conference on Computer Vision and Pattern Recognition (CVPR), demonstrating its significant research value and practical significance.

[0003] Currently, image dehazing algorithms can be broadly categorized into model-based parameter estimation and model-free image enhancement methods. Early methods, based on atmospheric scattering models, estimated transmittance and atmospheric light using physical priors such as dark channels, bright channels, and color attenuation to restore haze-free images. While these methods achieved effective dehazing, the transmittance estimates were often biased. With the advent of convolutional neural networks (CNNs), such as DehazeNet, AOD-Net, and DCPDN, ​​deep learning was employed to estimate parameters. While this improved parameter estimation accuracy, the high number of parameters and lack of rich features led to poor dehazing performance. Among model-free image enhancement methods, early methods such as histogram equalization, Rentinex, and image filtering restored haze-free images. Similarly, with the advent of deep learning, methods such as DMPHN and FFA-Net, which directly learn the end-to-end mapping relationship between hazy and haze-free images, have become one of the most mainstream dehazing methods.

[0004] Zhang et al. published "Deep Stacked Hierarchical Multi-patch Network for Image Deblurring" at the CVPR conference, proposing a deep multi-patch hierarchical network (DMPHN) for image deblurring. This approach divides the image into different blocks and layers, extracting features from a single layer using an encoder-decoder structure composed of multi-scale residual blocks. Residual connections are used between layers to aggregate feature information. Considering the non-uniform characteristics of haze and the similarities between image dehazing and deblurring, DasSD et al. published "Fast Deep Multi-patch Hierarchical Network for Non-nhomogeneous Image Dehazing" by reducing the number of layers and optimizing the loss function. They proposed a fast deep multi-patch dehazing network (FDMPHN) and a fast deep multi-scale hierarchical network (FDMSHN). Chinese patent publications CN 112767283A and CN 114972076 A, based on the DMPHN's block and layering concepts, invented "A Non-uniform Image Dehazing Method Based on Multi-Image Block Partitioning" and "An Image Dehazing Method Based on Hierarchical Multi-Block Convolutional Neural Network." Both the paper and the patent mentioned above employ residual connections for cross-level fusion, failing to consider the issues of image channel information redundancy and insufficient attention to pixel-domain feature information. Yang Kun et al. published "Fast Non-uniform Image Dehazing Based on Multi-Patch and Multi-Scale Hierarchical Aggregation Networks," which aggregates DMPHN and DMSHN and utilizes channel attention, dilated convolution, and frequency domain loss optimization. However, this inevitably increases algorithmic complexity, resulting in a 76% increase in runtime.

[0005] In summary, while numerous image dehazing methods exist, most focus on removing uniform haze. However, in real-world environments, haze is non-uniform, resulting in poor results. Furthermore, multi-patch methods for removing non-uniform haze directly fuse encoding and decoding results with corresponding blocks between layers, lacking effective utilization of encoding-decoding information and insufficient feature fusion, resulting in suboptimal color, brightness, and edge detail in the dehazed image. Summary of the Invention

[0006] In order to solve the problem that the existing technology has insufficient hierarchical feature fusion capability, resulting in poor image dehazing effect, the present invention provides a multi-patch dehazing method based on dual-attention hierarchical feature fusion (DAMPHN). By streamlining the encoder-decoder structure and adding a dual-attention hierarchical feature fusion module (DA) consisting of channel and pixel attention between different levels, the algorithm complexity is not increased.

[0007] In order to achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0008] A multi-patch dehazing method based on dual-attention level feature fusion includes the following steps:

[0009] Step 1: Obtain a training sample dataset: Obtain an existing clear / hazy dataset with non-uniform haze characteristics and perform data preprocessing according to training requirements.

[0010] Step 2: Build the DAMPHN network model: DAMPHN is a multi-layer network model consisting of an encoder module, a decoder module, and a dual-attention feature fusion module. DAMPHN is divided into i layers, where i = 1, 2, or 3, and each layer processes 4, 2, or 1 image blocks, respectively. Each layer corresponds to an encoder-decoder module. When fusing cross-layer information, the encoder output of the current layer and the encoder output of the next layer, as well as the decoder output of the current layer and the input image of the next layer, are fed into the dual-attention feature fusion module.

[0011] Step 3: Select reconstruction loss L r , Perceptual Loss L p and the total variational loss L tv Linear combination as loss function, L = λ r L r +λ p L p +λ tv L tv .

[0012] Step 4: Train the DAMPHN network model: First, set the training parameters and initialize the network model parameters. Then, take the jth block of the i-th level of the foggy image I, specifically denoted as I i,j Then, the first layer divides I into 4 blocks vertically and horizontally, denoted as U 1,1 、U 1,2 、U 1,3 、U 1,4 ; The second layer divides I into two blocks vertically, represented as I 2,1 , I 2,2 ; The third layer takes I directly as input, denoted as U 3,1 After the block segmentation and grading are completed, they are sent to DAMPHN training, and the parameters are optimized through the loss function to obtain the best dehazing model.

[0013] Furthermore, the decoder module in step 2 includes: a residual module, a deconvolution layer, a residual module, a deconvolution layer, a residual module, and a convolution layer, connected in sequence. The encoder module extracts feature information of the input image, and the decoder module reconstructs the image using the feature information obtained by the encoder module. The encoder module results of the current level and the encoder module results of the next level, and the decoder module results of the current level and the input image of the next level are respectively input into the dual-attention level feature fusion module for feature fusion, and the feature output is used to guide network training.

[0014] Furthermore, the dual-attention-level feature fusion module in step 2 includes: a channel attention module, a pixel attention module, and a convolution layer. After the convolution of the pixel attention module, the features are fused by adding the results of the channel attention module and the pixel attention module point by point.

[0015] The channel attention module is used to calibrate the feature correspondence of the channel domain and suppress poor or redundant features.

[0016] Assume that the feature map of the current level is F C ∈R H×W×C , the feature map of the previous level is F U ∈R H×W×C First, the feature maps are concatenated to obtain the aggregate feature map F, which is expressed as F = cat[F C ,F U ].

[0017] After adaptive global average pooling, it is expressed as: The feature map is transformed from C×H×W to C×1×1, and then passes through convolution-ReLU activation function-convolution-Sigmoid activation function in sequence to obtain the weight values ​​of different channels.

[0018] Expressed as:

[0019] The network structure of the pixel attention module includes: convolution, ReLU activation function, convolution, Sigmoid activation function, which is specifically expressed as:

[0020] After channel attention and pixel attention, the convolution is performed and element-wise added to make up for the information lost in the weighting of this dual attention layer. The specific expression is:

[0021] Furthermore, in step 3, the reconstruction loss is calculated to obtain the dehazed image J n The mean absolute error and mean square error between the clean image J are expressed as:

[0022] The perceptual loss is expressed as in, Represents the feature extractor corresponding to the convolutional layer of the pre-trained model VGG16 network.

[0023] Total variation loss L tv Expressed as in and To obtain the gradient matrix of the image in the horizontal and vertical directions respectively.

[0024] Furthermore, the specific steps of step 4 are as follows:

[0025] First, set the parameters and training strategy. Use the Adam optimizer, set the initial learning rate to 1-e4, and train for 10 cycles with a decay rate of 0.1 to adjust the learning rate. The parameter λ of the loss function is r ,λ1,λ2,λ p ,λ tv Set to 1, 0.6, 0.4, 6e-3, and 2e-8 respectively.

[0026] Secondly, the network training is carried out. Since each layer is composed of a pair of encoder-decoder, they can be represented as Enc i and Dec i . The input image I 1,j After passing through the encoder and DA modules in sequence, the encoding feature Q can be obtained i,j . It can be specifically expressed as:

[0027]

[0028] After the DA module and decoder, the local feature output J of each level can be obtained. i,j In particular, DAMPHN extracts features from the local to the whole idea, and its J 3,1 It represents the final dehazed image. It can be expressed as:

[0029]

[0030] Finally, the model is gradually optimized through the loss function.

[0031] Compared with the prior art, the advantages of the present invention are:

[0032] By streamlining the encoder-decoder structure and adding a dual-attention hierarchical feature fusion module consisting of channel and pixel attention between different layers, this approach addresses the problem of insufficient hierarchical feature fusion capabilities in existing technologies, which leads to poor image dehazing performance, while maintaining minimal algorithm complexity. This method achieves optimal dehazing results, with superior color and detail restoration, resulting in optimal dehazing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall structure diagram of the multi-patch dehazing network (DAMPHN) based on dual-attention level fusion according to an embodiment of the present invention;

[0034] Figure 2 is a network structure diagram of an encoder-decoder module according to an embodiment of the present invention;

[0035] Figure 3 1 is a network structure diagram of a dual-attention-level feature fusion module (DA) according to an embodiment of the present invention;

[0036] Figure 4 2. PSNR and SSIM test results of different dehazing methods according to the embodiment of the present invention;

[0037] Figure 5 This is a comparison of image dehazing results on the NH-HAZY and O-HAZY test sets according to an embodiment of the present invention, where:

[0038] (a) is a sample image with fog;

[0039] (b) is the dehazing result of the DCP model;

[0040] (c) is the dehazing result of the AOD-Net model;

[0041] (d) is the dehazing result of the FDMPHN model;

[0042] (e) is the defogging result of the model of the present invention;

[0043] (f) is the clear sample image corresponding to the foggy image. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0045] The present invention provides a multi-patch defogging method based on dual-attention level feature fusion, comprising the following steps:

[0046] Step 1: Obtain a sample training dataset;

[0047] We obtained three real, open-source haze / clear image pairs: Dense-HAZY, O-HAZY, and NH-HAZY. Dense-HAZY and NH-HAZY each have 55 haze / clear sample image pairs, with pairs 1-45 used for training, pairs 46-40 for validation, and pairs 51-55 for testing. O-HAZY has 45 haze / clear sample image pairs: pairs 1-35 for training, pairs 36-40 for validation, and pairs 41-45 for testing. Due to the small number of samples in these three datasets, we first divided the original image into 100 non-overlapping blocks. Next, we performed random rotations (0°, 90°, 180°, and 270°) for data augmentation.

[0048] Step 2: Build a multi-patch dehazing DAMPHN network model: The overall network structure of DAMPHN is as follows Figure 1 As shown, it includes an encoder-decoder module and a dual-attention level feature fusion module.

[0049] Figure 2 The figure shows the encoder-decoder module in the example of the present invention, in which the encoder includes 3 convolutional layers and 3 residual modules (convolution-ReLU activation function-convolution) connected in sequence. The decoder includes a residual module, a deconvolution layer, a residual module, a deconvolution layer, a residual module, and a convolution layer connected in sequence. The encoder extracts the feature information of the input image, and the decoder reconstructs the image using the feature information obtained by the encoder. The encoder result of the current level and the encoder result of the next level, and the decoder result of the current level and the input image of the next level are respectively input into the dual-attention level feature fusion module for feature fusion, and the feature output guides network training.

[0050] Figure 3 The figure shows a dual-attention-level feature fusion module in an example of the present invention, including a channel attention module, a pixel attention module, and a convolution layer. After the pixel attention convolution, the features are fused by adding the results of the channel attention and pixel attention point by point.

[0051] The channel attention module is used to calibrate the feature correspondence of the channel domain and suppress poor or redundant features. Assume that the feature map of the current level is F C ∈R H×W×C , the feature map of the previous level is F U ∈R H×W×C First, the feature maps are concatenated to obtain the aggregate feature map F, which is expressed as F = cat[F C ,F U ]. After adaptive global average pooling, it is expressed as: The feature map is transformed from C×H×W to C×1×1, and then goes through convolution-ReLU activation function-convolution-Sigmoid activation function in sequence to obtain the weight values ​​of different channels. It can be expressed as:

[0052] The pixel attention module focuses on areas with uneven pixel distribution in the image, making the fusion process more aware of areas with dense fog. Its network structure includes convolution-ReLU activation function-convolution-Sigmoid activation function, which can be expressed as follows:

[0053] After channel attention and pixel attention, the convolution is performed and element-wise added to make up for the information lost in the weighting of this dual attention layer. The specific expression is:

[0054] Step 3: Construct loss function;

[0055] Select the reconstruction loss L r , Perceptual Loss L p and the total variational loss L tv Linear combination as loss function, L = λ r L r +λ p L p +λ tv L tv .

[0056] The reconstruction loss is calculated for the dehazed image J n The mean absolute error and mean square error between the image and the clear image J can be expressed as:

[0057] Perceptual loss L p It can be expressed as in, Represents the feature extractor corresponding to the convolutional layer of the pre-trained model VGG16 network.

[0058] Total variation loss L tv Expressed as in and To obtain the gradient matrix of the image in the horizontal and vertical directions respectively.

[0059] Step 4: Train the DAMPHN network model

[0060] First, set the parameters and training strategy. Use the Adam optimizer, set the initial learning rate to 1-e4, and train for 10 cycles with a decay rate of 0.1 to adjust the learning rate. The parameter λ of the loss function is r ,λ1,λ2,λp ,λ tv Set to 1, 0.6, 0.4, 6e-3, and 2e-8 respectively.

[0061] Secondly, the network training is carried out. Since each layer is composed of a pair of encoder-decoder, they can be represented as Enc i and Dec i . The input image I 1,j After passing through the encoder and DA modules in sequence, the encoding feature Q can be obtained i,j . It can be specifically expressed as:

[0062]

[0063] After the DA module and decoder, the local feature output J of each level can be obtained. i,j In particular, DAMPHN extracts features from the local to the whole idea, and its J 3,1 It represents the final dehazed image. It can be expressed as:

[0064]

[0065] Finally, the model is gradually optimized through the loss function, and a total of 8 model files are obtained (6 codecs and 2 layer fusions).

[0066] Step 5: Verification test

[0067] The experimental environment of this example is NVIDIA RTX3090 (24GB). In order to more accurately evaluate the effectiveness of the DAMPHN network proposed in this invention, this model was quantitatively and qualitatively compared with DCP, AOD-Net, and FDMPHN after 200 rounds of training.

[0068] In the quantitative evaluation, first, the results of each round of training are as follows: Figure 4 shown. Figure 4 In the figure, the columns from left to right represent the test results on the Dense-HAZY, O-HAZY, and NH-HAZY datasets, and the rows from top to bottom represent the PSNR and SSIM evaluation parameters. Figure 4 As can be seen from the results, DAMPHN achieves the best PSNR and SSIM values ​​on both the non-uniform fog datasets, O-HAZY and NH-HAZY. On the uniform dense fog dataset, Dense-HAZY, it achieves the highest PSNR and AOD-Net the highest SSIM, but these values ​​decrease with increasing training times during later training. Furthermore, based on the above experiments, the average processing time per image (ART / s) was further quantified and analyzed. The specific experimental results are shown in Table 1.

[0069] Table 1 Quantitative comparison experimental results on three datasets

[0070]

[0071]

[0072] The bold and underlined values ​​in Table 1 indicate the optimal values. Across three public datasets, the PSNR of our method is, on average, 3.72dB, 1.06dB, and 0.3dB higher than those of DCP, AOD-Net, and FDMPHN, respectively. Similarly, the SSIM scores of our method are, on average, 0.0185, 0.0212, and 0.0594 higher than those of the other three methods, respectively.

[0073] In the qualitative evaluation, the results of the real non-uniform haze datasets O-HAZY and NH-HAZY are taken as examples. The corresponding foggy images are Figure 5 (a), the defogging result of this example is Figure 5 (e) From Figure 5 As can be seen, DCP dehazing results in severe color distortion and is largely ineffective in dense fog areas. AOD-Net dehazing also exhibits significant color distortion and is also largely ineffective in dense fog areas. FDMPHN dehazing results inadequately restore detailed features in dense fog areas. This method achieves the best dehazing results, with superior color and detail restoration, resulting in the most effective dehazing results.

[0074] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-patch dehazing method based on dual-attention level feature fusion, characterized by: The following steps are involved: Step 1: Obtain a training sample dataset: Obtain an existing clear / hazy dataset with non-uniform haze characteristics and perform data preprocessing according to training requirements; Step 2: Build the DAMPHN network model: DAMPHN is a multi-layer network model, including an encoder module, a decoder module, and a dual-attention level feature fusion module; DAMPHN is divided into i levels, where i = 1, 2, 3, where each level processes 4, 2, and 1 blocks of images respectively; each level has an encoder-decoder module corresponding to it; In cross-level information fusion, the encoder result of the current level and the encoder result of the next level, as well as the decoder result of the current level and the input image of the next level are input into the dual-attention level feature fusion module (DA); The dual-attention-level feature fusion module includes: a channel attention module, a pixel attention module, and a convolution layer. After the convolution of the pixel attention module, the features are fused by adding the results of the channel attention module and the pixel attention module point by point. The channel attention module is used to calibrate the feature correspondence of the channel domain and suppress poor or redundant features; Assume that the feature map of the current level is F C ∈R H×W×C , the feature map of the previous level is F U ∈R H×W×C ; First, the feature maps are concatenated to obtain the aggregate feature map F, which is expressed as F = cat[F C ,F U ]; After adaptive global average pooling, it is expressed as: The feature map is transformed from C×H×W to C×1×1, and then passes through convolution-ReLU activation function-convolution-Sigmoid activation function in sequence to obtain the weight values ​​of different channels; Expressed as: The network structure of the pixel attention module includes: convolution, ReLU activation function, convolution, Sigmoid activation function, which is specifically expressed as: After channel attention and pixel attention, the convolution is performed and element-wise added to make up for the information lost in the weighting of this dual attention layer. It is specifically expressed as: F out =Conv(F PA ,k=3)⊕F PA ⊕F CA ⊕F; Step 3: Select reconstruction loss L r , Perceptual Loss L p and the total variational loss L tv Linear combination as loss function, L = λ r L r +λ p L p +λ tv L tv ; Step 4: Train the DAMPHN network model: First, set the training parameters and initialize the network model parameters; then, train the jth block of the i-th level of the foggy image I, specifically denoted as I i,j ; Then, the first layer divides I into 4 blocks vertically and horizontally, represented as I 1,1 , I 1,2 , I 1,3 , I 1,4 ; The second layer divides I into two blocks vertically, represented as I 2,1 , I 2,2 ; The third layer takes I directly as input, expressed as I 3,1 After the block division and classification are completed, they are sent to DAMPHN training, and the parameters are optimized through the loss function to obtain the best dehazing model; Step 4: The specific steps are as follows: First, set the parameters and training strategy; use the Adam optimizer, set the initial learning rate to 1-e4, train for 10 cycles and adjust the learning rate with a decay rate of 0.1; the parameter λ of the loss function r ,λ1,λ2,λ p ,λ tv Set to 1, 0.6, 0.4, 6e-3, 2e-8 respectively; Secondly, the network training is carried out. Since each layer is composed of a pair of encoder-decoder, they can be represented as Enc i and Dec i ; Input image I 1,j After passing through the encoder and DA modules in sequence, the encoding feature Q can be obtained i,j ; Specifically, it can be expressed as: After the DA module and decoder, the local feature output J of each level can be obtained. i,j ; DAMPHN feature extraction from the local to the whole idea, its J 3,1 It represents the final dehazed image; it can be specifically expressed as: Finally, the model is gradually optimized through the loss function.

2. The multi-patch dehazing method based on dual-attention-level feature fusion according to claim 1, characterized in that: In step 2, the decoder module includes: a residual module, a deconvolution layer, a residual module, a deconvolution layer, a residual module, and a convolution layer connected in sequence; the feature information of the input image is extracted through the encoder module, and the decoder module reconstructs the image using the feature information obtained by the encoder module; the encoder module result of the current level and the encoder module result of the next level, and the decoder module result of the current level and the input image of the next level are respectively input into the dual-attention level feature fusion module for feature fusion, and the feature output is used to guide network training.

3. The multi-patch dehazing method based on dual-attention-level feature fusion according to claim 1, characterized in that: In step 3, the reconstruction loss is calculated as the dehazed image J n The mean absolute error and mean square error between the clean image J are expressed as: The perceptual loss is expressed as in, Represents the feature extractor corresponding to the convolutional layer of the pre-trained model VGG16 network; Total variation loss L tv Expressed as in and To obtain the gradient matrix of the image in the horizontal and vertical directions respectively.

Images