An image dehazing method based on multi-feature attention mechanism

Abstract

The present invention discloses an image defogging method based on a multi-feature attention mechanism, which relates to the technical field of image defogging. Based on U-Net, a multi-feature attention module is integrated to synthesize feature information of different scales and fuse shallow and deep features, thereby improving feature expression capabilities, especially in retaining edge information. Skip connections are introduced to alleviate the gradient vanishing problem. The multi-feature attention module can unify the feature resolutions of different levels and use a maximum pooling algorithm to retain edge details. It consists of a channel attention module, a pixel attention module and a skip connection. The channel attention judges the importance of the channel and performs weighted processing, while the pixel attention focuses on the pixel feature assignment weights. The present invention overcomes the limitations of traditional defogging technology, utilizes the advantages of deep learning, improves processing efficiency and image quality, is applicable to foggy images in complex environments, effectively reduces feature information loss, and improves image processing accuracy and detail restoration effects.

Description

Technical Field

[0001] The present invention relates to the technical field of image defogging in computer vision, and in particular to an image defogging method based on a multi-feature attention mechanism. Background Art

[0002] Foggy weather creates numerous tiny water droplets that scatter or absorb light passing through them, causing color distortion, reduced contrast, and blurred details in captured images, impacting image quality and usability. This phenomenon is particularly prominent in areas such as surveillance, traffic management, autonomous vehicles, and outdoor image acquisition. Fog not only misleads visual perception systems but can also affect subsequent image analysis and decision-making. For example, in autonomous driving systems, foggy image processing has become a key technology for ensuring driving safety.

[0003] There are two traditional dehazing techniques: one based on image enhancement and the other on physical models. Image enhancement methods improve image quality by adjusting image contrast and brightness, but often ignore the physical causes of fog, resulting in unnatural dehazing results. While physical model-based methods take into account atmospheric scattering models, they typically require the estimation of complex environmental parameters, resulting in high computational complexity and significantly reduced dehazing effectiveness when parameter estimates are inaccurate. In recent years, deep learning methods have been widely used for image dehazing due to their powerful feature extraction capabilities. These methods can automatically learn the distribution characteristics of fog and effectively remove fog based on this information, significantly improving processing efficiency and image quality. However, current image dehazing methods are often limited by their reliance on estimated parameters, particularly when dealing with foggy images in complex, real-world environments. Summary of the Invention

[0004] To address these issues, we designed an image dehazing method based on a multi-feature attention mechanism. U-Net, with its unique skip connections and optimization of fully convolutional networks, has achieved remarkable results in medical image segmentation tasks. This method expands upon U-Net by integrating a multi-feature attention module, enabling the network to synthesize feature information at different scales and effectively fuse shallow and deep features. This design significantly enhances the network's ability to express features, particularly in preserving image details such as edges.

[0005] The main process of its information processing is as follows:

[0006] S1.1. The input image is extracted through the initial feature extraction layer to obtain a residual block, and the residual block and feature extraction layer are saved through skip connections;

[0007] S1.2. Downsample the obtained residual block through a convolution operation to obtain a downsampling layer, then perform feature extraction on the downsampling layer to obtain a feature extraction layer, and save the feature extraction layer through a skip connection;

[0008] S1.3. Extract features from the feature extraction layer through the feature attention module, and integrate the residual blocks saved in the skip connection to obtain the deep residual blocks, and save the residual blocks through the skip connection;

[0009] S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated;

[0010] S1.5. Extract features from the deepest feature extraction layer through the feature attention module to obtain the upsampling layer;

[0011] S1.6. Extract features from the upsampling layer to obtain a residual attention layer. Use the multi-feature attention module to integrate the feature extraction layer saved in the skip connection, and concatenate the integration result with the residual attention layer to generate the upper upsampling layer.

[0012] S1.7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layers saved in the jump connection through the multi-feature attention module, and concatenate the integration results with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

[0013] During image processing, the vanishing gradient phenomenon becomes more pronounced as network depth increases. This problem can lead to better updates for weights near the output layer, while updates for weights in deeper layers become slower, impacting network training efficiency. The introduction of skip connections, which transfer information through direct connections between network layers, effectively alleviates the vanishing gradient problem. This design allows the network to bypass some layers and directly use the output of one layer as the input of adjacent layers, thereby enhancing feature transfer efficiency and network training performance. However, skip connections have certain limitations in addressing the vanishing gradient problem, as they can only link feature maps with consistent resolution.

[0014] To solve this problem, the present invention provides an innovative multi-feature attention module. The data processing process is as follows:

[0015] Pooling and integration of feature extraction layers saved in skip connections;

[0016] The integrated data is processed by the pixel attention module to obtain intermediate data;

[0017] The intermediate data is added element by element to the feature extraction layer to obtain the processed data.

[0018] This module can unify features at different levels to the same resolution, achieving effective fusion of shallow and deep features.

[0019] Preferably, the above pooling processing method is a maximum pooling algorithm.

[0020] In theory, there are two main sources of feature extraction errors:

[0021] Due to the limitation of neighborhood size, the estimation results vary greatly;

[0022] Due to the errors in the convolutional network parameters, the estimated mean value is biased.

[0023] The average pooling algorithm effectively reduces the first type of error by finding the average value of each feature point in each region, while also better preserving the image's background information. The maximum pooling algorithm, on the other hand, reduces the second type of error by selecting the maximum value of feature points within a region, while also placing greater emphasis on the image's texture features. To address the difficulty of obtaining edge information during image dehazing, the multi-feature attention module proposed in this paper specifically employs the maximum pooling algorithm to more effectively preserve image edge details.

[0024] Furthermore, the feature attention module consists of a channel attention module, a pixel attention module and a skip connection.

[0025] Furthermore, the process of data processing in the channel attention module is as follows:

[0026] Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C Feature map g c The formula is as follows:

[0027] (1.1)

[0028] In formula (1.1) H Represents the vertical size of the image, W Represents the horizontal size of the image, X c (i, j) Indicates the c Channel image X In position (i, j) The pixel size at H p represents the global pooling method, F c Represents the input image.

[0029] The feature map obtained g cThe compressed convolution operation is performed in proportion to the number of channels, and then the convolution operation is used to restore it to the original number of channels. Finally, the generated value is mapped to the range of 0 to 1 using the Sigmoid function to obtain the attention weight corresponding to each channel. CA c The formula is as follows:

[0030] (1.2)

[0031] In formula (1.2), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0032] Input data F c and attention weights CA c Perform element-by-element multiplication to obtain the processed data. The formula is as follows:

[0033] (1.3)

[0034] In formula (1.3), F c Represents the input feature map, CA c represents the weight of the channel, F c * Represents the data that has been processed. Represents element-wise multiplication operation.

[0035] The channel attention module realizes the judgment of the importance of channels and weights them according to their importance, thereby making more effective use of feature information.

[0036] Furthermore, the process of pixel attention module processing data is as follows:

[0037] The dimension is H×W×C Data F * Input into a convolution layer with ReLU linear rectification function, the dimension is H×W×1 Feature map of

[0038] Then, after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained. PA The formula is as follows:

[0039] (1.4)

[0040] In formula (1.4), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0041] The data to be entered F * and attention weights PA Perform element-by-element multiplication to obtain the processed data. The formula is as follows:

[0042] (1.5)

[0043] In formula (1.5), The data that has been processed.

[0044] In real environments, fog and haze are often unevenly distributed. This mechanism enables the entire network to focus more on the characteristics of pixel information, such as high-frequency regions and dense haze pixels in an image. The model can automatically focus on certain pixels in the image, assigning different attention weights based on their importance to the task, and then use these weights to process the pixels, thereby improving the model's understanding and processing capabilities.

[0045] The present invention effectively reduces feature information loss and improves the accuracy of image processing tasks and detail recovery effects through the combined effect of jump connections and the above-mentioned multiple feature attention modules. Utilizing multiple feature attention modules, the effective fusion of features at multiple levels is achieved, breaking through the reliance of a single feature on pixel weight information. Compared to calculating pixel weight information based solely on single-layer features, this method uses a strategy of calculating pixel weight information using multiple layers of features, thereby reducing information loss during the model encoding and downsampling process.

[0046] A method for defogging an image is also provided, comprising the following steps:

[0047] Acquire images with fog through camera equipment;

[0048] The image is dehazed using the above image dehazing method based on the multi-feature attention mechanism;

[0049] Obtain a dehazed image.

[0050] The advantages of the present invention are:

[0051] 1. This invention expands upon U-Net by integrating a multi-feature attention module. This module can synthesize feature information at different scales and effectively fuse shallow and deep features. It excels at preserving image details (such as edge information) and enhances the network's ability to express features.

[0052] 2. The introduction of a skip connection mechanism, which transfers information directly between network layers, effectively alleviates the vanishing gradient problem, enhances feature transfer efficiency, and improves network training performance. Although this mechanism has certain limitations in associating feature maps with consistent resolution, it has a positive effect on network training overall.

[0053] 3. A particular advantage of this invention lies in its multi-feature attention module's optimized skip connection mechanism, which unifies features at different levels to the same resolution, effectively fusing shallow and deep features. Furthermore, the use of a maximum pooling algorithm effectively preserves image edge details, addressing the difficulty in obtaining edge information during image dehazing.

[0054] 4. This method overcomes the limitations of traditional dehazing technologies (e.g., image enhancement-based methods that ignore the physical causes of fog, and physical model-based methods that suffer from problems with parameter estimation and large computational complexity). By leveraging the powerful feature extraction capabilities of deep learning and combining it with its own design, it significantly improves processing efficiency and image quality, making it particularly suitable for processing foggy images in complex real-world environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart of this method.

[0056] Figure 2 Flowchart of the feature attention module.

[0057] Figure 3 Flowchart of the multi-feature attention module (MF1).

[0058] Figure 4 Flowchart of the multi-feature attention module (MF2).

[0059] Figure 5 Flowchart of the multi-feature attention module (MF3). DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0061] like Figure 1 As shown in the figure, the non-straight arrows represent skip connections. The image defogging method based on the multi-feature attention mechanism disclosed in the present invention has the following process:

[0062] S1.1. The input image is extracted through the initial feature extraction layer to obtain a residual block, and the residual block and feature extraction layer are saved through skip connections;

[0063] S1.2. Downsample the obtained residual block through a convolution operation to obtain a downsampling layer, then perform feature extraction on the downsampling layer to obtain a feature extraction layer, and save the feature extraction layer through a skip connection;

[0064] S1.3. Extract features from the feature extraction layer through the feature attention module, and integrate the residual blocks saved in the skip connection to obtain the deep residual blocks, which are saved through the skip connection.

[0065] S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated;

[0066] S1.5. Extract features from the deepest feature extraction layer through the feature attention module to obtain the upsampling layer;

[0067] S1.6. Extract features from the upsampling layer to obtain a residual attention layer. Use the multi-feature attention module to integrate the feature extraction layer saved in the skip connection, and concatenate the integration result with the residual attention layer to generate the upper upsampling layer.

[0068] S1.7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layers saved in the jump connection through the multi-feature attention module, and concatenate the integration results with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

[0069] The above process of data processing through the feature attention module is as follows Figure 2 As shown:

[0070] In the picture represents the Sigmoid function, Represents element-wise multiplication.

[0071] The feature attention module consists of a channel attention module (CA), a pixel attention module (PA), and a skip connection. The non-straight arrows in the figure represent skip connections. The specific process is as follows:

[0072] Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C Feature map g c The formula is as follows:

[0073] (1.1)

[0074] In formula (1.1) H Represents the vertical size of the image, W Represents the horizontal size of the image, X c (i, j) Indicates thec Channel image X In position (i, j) The pixel size at H p represents the global pooling method, F c Represents the input image.

[0075] The feature map obtained g c The compressed convolution operation is performed in proportion to the number of channels, and then the convolution operation is used to restore it to the original number of channels. Finally, the generated value is mapped to the range of 0 to 1 using the Sigmoid function to obtain the attention weight corresponding to each channel. CA c The formula is as follows:

[0076] (1.2)

[0077] In formula (1.2), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0078] Input data F c and attention weights CA c Perform element-by-element multiplication and obtain the dimension of the completed processing: H×W ×C The formula is as follows:

[0079] (1.3)

[0080] In formula (1.3), F c Represents the input feature map, CA c represents the weight of the channel, F c * Represents the data that has been processed. Represents element-wise multiplication operation.

[0081] The dimension is H×W×C Data F * Input into a convolution layer with ReLU linear rectification function, the dimension is H×W×1 Feature map of

[0082] Then, after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained. PA The formula is as follows:

[0083] (1.4)

[0084] In formula (1.4), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0085] The data to be entered F * and attention weights PA Perform element-by-element multiplication to obtain the processed data. The formula is as follows:

[0086] (1.5)

[0087] In formula (1.5), The data that has been processed.

[0088] Since jump connections can only associate feature maps with the same resolution, they have certain limitations in solving the vanishing gradient problem. To solve this problem, the present invention provides a multi-feature attention module. Figures 3 to 5 The flowchart of the multi-feature attention module MFA1~MFA3 is shown in Figure 5 Taking the multi-feature attention module MF3 as an example, the data processing process is described in detail:

[0089] In the picture Indicates the integration operation, represents the Sigmoid function, represents element-wise multiplication, Represents element-wise addition.

[0090] This layer integrates the features of the first to third layers that have undergone pooling (Max Pool) with the features of the fourth layer. The integrated features are then processed by the pixel attention module to generate weighted outputs. These weights are multiplied and added element-by-element with the residual block of the fourth layer, so that the features of the lower-level first, second, and third layers can guide the direction of the fourth-layer residual block in pixel weight calculation. This processing ensures that when calculating pixel weights, the fourth-layer residual block can comprehensively consider information from deep and shallow layers. In addition, combined with the use of jump connections, these operations jointly promote the effective propagation of features and further alleviate the problem of gradient disappearance in many aspects. A similar processing flow is also used in the multi-feature attention modules MFA1 and MFA2.

[0091] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0092] Although the above describes the specific implementation methods of the present invention, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An image dehazing method based on a multi-feature attention mechanism, characterized in that: Expanding on the U-Net model and integrating a multi-feature attention module, the following steps are involved: S1.

1. Extract features from the input image through the initial feature extraction layer to obtain a residual block, and save the residual block and the feature extraction layer through a skip connection; S1.

2. Downsampling the obtained residual block through a convolution operation to obtain a downsampling layer, then performing feature extraction on the downsampling layer to obtain a feature extraction layer, and saving the feature extraction layer through the skip connection; S1.

3. Extract features from the feature extraction layer using a feature attention module, and integrate the residual blocks stored in the jump connection to obtain the deep residual blocks, and store the residual blocks through the jump connection; S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated; S1.

5. Extract features from the deepest feature extraction layer using the feature attention module to obtain an upsampling layer; S1.

6. Extract features from the upsampling layer to obtain a residual attention layer, integrate the feature extraction layer stored in the skip connection through a multi-feature attention module, and concatenate the integration result with the residual attention layer to generate the upper upsampling layer; S1.

7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layer saved in the jump connection through the multi-feature attention module, and splice the integration result with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

2. The image defogging method based on a multi-feature attention mechanism according to claim 1, characterized in that: The integration process of the multi-feature attention module described in steps S1.6 and S1.7 is as follows: S2.

1. Pooling and integrating the feature extraction layers stored in the skip connection; S2.2, processing the integrated data through the pixel attention module to obtain intermediate data; S2.

3. Add the intermediate data to the deepest feature extraction layer element by element to obtain the processed upper residual attention layer.

3. The image defogging method based on a multi-feature attention mechanism according to claim 2, characterized in that: The pooling process in step S2.1 is performed by a maximum pooling algorithm.

4. The image defogging method based on a multi-feature attention mechanism according to claim 1, characterized in that: The feature attention module consists of a channel attention module, a pixel attention module and a skip connection.

5. The image defogging method based on a multi-feature attention mechanism according to claim 4, characterized in that: The process of processing data through the channel attention module is as follows: S3.

1. Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C Feature map of S3.

2. Perform a compressed convolution operation on the feature map obtained in step S3.1 in proportion to the number of channels, and then restore it to the original number of channels through the convolution operation. Finally, use the Sigmoid activation function to map the generated values to the range of 0 to 1 to obtain the attention weight corresponding to each channel; S3.

3. Multiply the data in step S3.1 and the attention weight obtained in step S3.2 element by element to obtain the processed data.

6. The image defogging method based on a multi-feature attention mechanism according to any one of claims 2 and 4, characterized in that: The process of processing data through the pixel attention module is as follows: S4.1、dimension H×W×C The data is input into a convolution layer with a ReLU linear rectification function to obtain a dimension of H×W×1 Feature map of S4.2, then after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained; S4.

3. Multiply the data in step S4.1 by the attention weight obtained in step S4.2 element by element to obtain the processed data.

7. An image defogging method, characterized in that: The following steps are involved: S5.

1. Acquire an image with fog by a camera; S5.

2. Defogging the image using the image defogging method based on a multi-feature attention mechanism according to any one of claims 1 to 6; S5.

3. Obtain a dehazed image.

Images