Image defogging model and defogging method based on multi-feature attention mechanism

Abstract

The invention discloses an image defogging model and defogging method based on a multi-feature attention mechanism. A multi-feature attention module is integrated on the basis of U-Net, feature information of different scales is integrated, shallow and deep features are fused, the feature expression ability is improved, and particularly, the performance is excellent in the aspect of reserving edge information. And introducing jump connection to relieve the gradient disappearance problem. The multi-feature attention module can unify different levels of feature resolutions, and edge details are reserved by adopting a maximum pooling algorithm. The method is composed of a channel attention module, a pixel attention module and jump connection, channel attention evaluation channel importance weighting processing is carried out, and pixel attention focuses on pixel feature distribution weight. The method overcomes the limitation of a traditional defogging technology, utilizes the advantages of deep learning, improves the processing efficiency and the image quality, is suitable for a foggy image in a complex environment, effectively reduces the loss of feature information, and improves the image processing accuracy and the detail recovery effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image defogging in computer vision, and in particular to an image defogging model and a defogging method based on a multi-feature attention mechanism. Background Art

[0002] Foggy weather will form many tiny water droplets, which will scatter or absorb the light passing through them, causing the captured images to have color distortion, reduced contrast and blurred details, affecting the quality and usability of the images. This phenomenon is particularly prominent in the fields of monitoring, traffic management, driverless cars and outdoor image acquisition. Haze will not only mislead the judgment of the visual perception system, but may also affect the subsequent image analysis and decision-making process. For example, in the autonomous driving system, foggy image processing has become one of the key technologies to ensure driving safety.

[0003] There are two traditional defogging technologies, one based on image enhancement and the other based on physical models. Image enhancement methods improve image quality by adjusting the contrast and brightness of images, but often ignore the physical causes of fog, resulting in unnatural defogging effects. Although physical model-based methods take into account the atmospheric scattering model, they usually require the estimation of complex environmental parameters, which is computationally intensive and will greatly reduce the defogging effect when parameter estimation is inaccurate. In recent years, deep learning methods have been widely used in image defogging due to their powerful feature extraction capabilities. These methods can automatically learn the distribution characteristics of fog and perform effective defogging on this basis, significantly improving processing efficiency and image quality. However, current image defogging methods are usually limited to relying on estimated parameters, especially when dealing with foggy images in complex real-world environments. Summary of the invention

[0004] To solve the above problems, the present invention designs an image dehazing model based on a multi-feature attention mechanism. U-Net has achieved remarkable results in medical image segmentation tasks due to its unique skip connection and optimization of the fully convolutional network. The present invention expands on the basis of U-Net and integrates a multi-feature attention module, so that the network can integrate feature information of different scales and effectively fuse shallow and deep features. This design significantly improves the network's ability to express features, especially in retaining image details, such as edge information.

[0005] The main process of information processing is as follows: S1.1, the input image is extracted through the initial feature extraction layer to obtain a residual block, and the residual block and the feature extraction layer are saved through a jump connection; S1.2, downsampling the obtained residual block through a convolution operation to obtain a downsampling layer, and then extracting features from the downsampling layer to obtain a feature extraction layer, and saving the feature extraction layer through a skip connection; S1.3, extract features from the feature extraction layer through the feature attention module, and integrate the residual blocks saved in the jump connection to obtain the deep residual blocks, and save the residual blocks through the jump connection; S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated; S1.5, extract features from the deepest feature extraction layer through the feature attention module to obtain an upsampling layer; S1.6. Extract features from the upsampling layer to obtain a residual attention layer, integrate the feature extraction layer saved in the jump connection through the multi-feature attention module, and concatenate the integration result with the residual attention layer to generate the upper upsampling layer; S1.7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layers saved in the jump connection through the multi-feature attention module, and concatenate the integration result with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

[0006] As the depth of the network increases during image processing, the vanishing gradient phenomenon becomes more prominent. This problem may cause the weights close to the output layer in the network to be updated well, while the weights in the deeper layers are updated slowly, affecting the training efficiency of the network. By introducing the skip connection mechanism, that is, transmitting information through direct connections between network layers, the problem of vanishing gradients is effectively alleviated. This design allows the network to bypass some layers and directly use the output of a layer as the input of the adjacent layer, thereby enhancing the feature transmission efficiency and the training performance of the network. Since skip connections can only associate feature maps with consistent resolution, they have certain limitations in solving the problem of vanishing gradients.

[0007] To solve this problem, the present invention provides an innovative multi-feature attention module. The data processing process is as follows: Pooling and integration of feature extraction layers saved in skip connections; The integrated data is processed by the pixel attention module to obtain intermediate data; The intermediate data is added element by element to the feature extraction layer to obtain the processed data.

[0008] This module can unify features at different levels to the same resolution, achieving effective fusion of shallow and deep features.

[0009] Preferably, the above pooling processing method is a maximum pooling algorithm.

[0010] In theory, there are two main sources of feature extraction errors: Due to the limitation of neighborhood size, the estimation results vary greatly; Due to errors in the convolutional network parameters, the estimated mean value is biased.

[0011] The average pooling algorithm can effectively reduce the first error by finding the average value of each feature point in each area, and can better retain the background information of the image. The maximum pooling algorithm reduces the second error by selecting the maximum value of the feature points in the area, and also pays more attention to the texture characteristics of the image. In view of the difficulty in obtaining edge information during image dehazing, the multi-feature attention module proposed in the present invention particularly adopts the maximum pooling algorithm to more effectively retain the edge details of the image.

[0012] Furthermore, the feature attention module consists of a channel attention module, a pixel attention module and a skip connection.

[0013] Furthermore, the process of data processing in the channel attention module is as follows: Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C Feature map g c The formula is as follows: (1.1) In formula (1.1) H Represents the vertical size of the image, W Represents the horizontal size of the image, X c (i, j) Indicates c Channel image X In Location (i, j) The pixel size at H p represents the global pooling method, F c Represents the input image.

[0014] The feature map obtained g c The compressed convolution operation is performed in proportion to the number of channels, and then restored to the original number of channels through the convolution operation. Finally, the generated value is mapped to the range of 0 to 1 using the Sigmoid function to obtain the attention weight corresponding to each channel. CA c The formula is as follows: (1.2) In formula (1.2), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0015] Enter the data F c and attention weights CA c Perform element-by-element multiplication to obtain the processed data. The formula is as follows: (1.3) In formula (1.3), F c Represents the input feature map, CA c represents the weight of the channel, F c * Represents the data that has been processed. Represents element-wise multiplication operation.

[0016] The channel attention module realizes the judgment of the importance of channels and weights them according to their importance, thereby making more effective use of feature information.

[0017] Furthermore, the process of pixel attention module processing data is as follows: The dimension is H×W×C Data F * Input into a convolutional layer with a ReLU linear rectification function, the dimension is H×W×1 The feature map of Then, after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained. PA The formula is as follows: (1.4) In formula (1.4), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0018] The data to be entered F * With attention weight PA Perform element-by-element multiplication to obtain the processed data. The formula is as follows: (1.5) In formula (1.5), The data for processing is completed.

[0019] In real environments, fog and haze are often unevenly distributed. This mechanism makes the entire network pay more attention to the characteristics of pixel information, such as high-frequency areas and dense haze pixels in the image. The model can automatically focus on certain pixels in the image, assign different attention weights according to the importance of these pixels to the task, and then use these weights to process the pixels, thereby improving the model's understanding and processing capabilities of the image.

[0020] The present invention effectively reduces feature information loss and improves the accuracy of image processing tasks and detail recovery effects through the combined effect of jump connections and the above-mentioned multiple feature attention modules. By using multiple feature attention modules, the effective fusion of features at multiple levels is achieved, breaking through the dependence of a single feature on pixel weight information. Compared with calculating pixel weight information based on a single layer of features alone, this method uses a strategy of calculating pixel weight information using multiple layers of features, thereby reducing the loss of information in the model during encoding and downsampling.

[0021] A method for defogging an image is also provided, comprising the following steps: Acquire images with haze through camera equipment; The image is input into the above-mentioned image dehazing model based on multi-feature attention mechanism for dehazing; Get the dehazed image.

[0022] The advantages of the present invention are: 1. This invention expands on U-Net and integrates a multi-feature attention module. It can integrate feature information of different scales and effectively fuse shallow and deep features. It performs well in preserving image details (such as edge information) and improves the network's ability to express features. 2. The introduction of the skip connection mechanism can effectively alleviate the gradient vanishing problem and enhance the feature transfer efficiency and network training performance by directly connecting the network layers. Although it has certain limitations in associating feature maps with consistent resolution, it has a positive effect on network training overall. 3. The special advantage of the present invention is that the multi-feature attention module optimizes the skip connection mechanism, which can unify the features of different levels to the same resolution, realize the effective fusion of shallow and deep features, and especially adopt the maximum pooling algorithm to more effectively retain the edge details of the image to cope with the problem that the edge information is difficult to obtain in the image dehazing process; 4. It overcomes the limitations of traditional dehazing technologies (such as methods based on image enhancement that ignore the physical causes of fog, and methods based on physical models that have problems with parameter estimation and large amounts of computation), and utilizes the powerful feature extraction capabilities of deep learning combined with its own design to significantly improve processing efficiency and image quality. It is particularly suitable for processing foggy images in complex environments in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the flow chart of this model.

[0024] Figure 2 Flowchart of the feature attention module.

[0025] Figure 3 Flowchart of the multi-feature attention module (MF1).

[0026] Figure 4 Flowchart of the multi-feature attention module (MF2).

[0027] Figure 5 Flowchart of the multi-feature attention module (MF3). DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with specific embodiments and drawings.

[0029] like Figure 1 As shown in the figure, the non-straight arrows represent skip connections. The image defogging model based on a multi-feature attention mechanism disclosed in the present invention has the following process: S1.1, the input image is extracted through the initial feature extraction layer to obtain a residual block, and the residual block and the feature extraction layer are saved through a jump connection; S1.2, downsampling the obtained residual block through a convolution operation to obtain a downsampling layer, and then extracting features from the downsampling layer to obtain a feature extraction layer, and saving the feature extraction layer through a skip connection; S1.3, extract features from the feature extraction layer through the feature attention module, and integrate the residual blocks saved in the jump connection to obtain the deep residual blocks, and save the residual blocks through the jump connection; S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated; S1.5, extract features from the deepest feature extraction layer through the feature attention module to obtain an upsampling layer; S1.6. Extract features from the upsampling layer to obtain a residual attention layer, integrate the feature extraction layer saved in the jump connection through the multi-feature attention module, and concatenate the integration result with the residual attention layer to generate the upper upsampling layer; S1.7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layers saved in the jump connection through the multi-feature attention module, and concatenate the integration result with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

[0030] The above process of data processing through the feature attention module is as follows Figure 2 As shown: In the picture represents the Sigmoid function, Represents element-wise multiplication.

[0031] The feature attention module consists of a channel attention module (CA), a pixel attention module (PA) and a skip connection. The non-straight arrows in the figure represent skip connections. The specific process is as follows: Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C Feature map g c The formula is as follows: (1.1) In formula (1.1) H Represents the vertical size of the image, W Represents the horizontal size of the image, X c (i, j) Indicates c Channel image X In Location (i, j) The pixel size at H p represents the global pooling method, F c Represents the input image.

[0032] The feature map obtained g c The compressed convolution operation is performed in proportion to the number of channels, and then restored to the original number of channels through the convolution operation. Finally, the generated value is mapped to the range of 0 to 1 using the Sigmoid function to obtain the attention weight corresponding to each channel. CA c The formula is as follows: (1.2) In formula (1.2), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0033] Enter the data F c and attention weights CA c Perform element-by-element multiplication and obtain the dimension of the processed H×W ×C The formula is as follows: (1.3) In formula (1.3), F c Represents the input feature map, CA c represents the weight of the channel, F c * Represents the data that has been processed. Represents element-wise multiplication operation.

[0034] The dimension is H×W×C Data F * Input into a convolutional layer with a ReLU linear rectification function, the dimension is H×W×1 The feature map of Then, after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained. PA The formula is as follows: (1.4) In formula (1.4), Conv represents the convolution operation, σ represents the Sigmoid function, δ Represents the ReLU function.

[0035] The data to be entered F * With attention weight PA Perform element-by-element multiplication to obtain the processed data. The formula is as follows: (1.5) In formula (1.5), The data for processing is completed.

[0036] Since skip connections can only associate feature maps with the same resolution, they have certain limitations in solving the gradient vanishing problem. To solve this problem, the present invention provides a multi-feature attention module. Figure 3~Figure 5 The flowchart of the multi-feature attention module MFA1~MFA3 is shown in Figure 5Taking the multi-feature attention module MF3 as an example, the data processing process is described in detail: In the figure Indicates the integration operation. represents the Sigmoid function, represents element-wise multiplication, Represents element-by-element addition.

[0037] This layer integrates the features of the first to third layers after pooling (Max Pool) with the features of the fourth layer. The integrated features are then processed by the pixel attention module to generate weighted outputs. These weights are element-by-element multiplication and addition operations with the residual blocks of the fourth layer, so that the features of the first, second, and third layers at the lower levels can guide the direction of the fourth layer residual blocks in the calculation of pixel weights. This processing ensures that when calculating pixel weights, the fourth layer residual blocks can comprehensively consider information from deep and shallow layers. In addition, combined with the use of jump connections, these operations jointly promote the effective propagation of features and further alleviate the problem of gradient disappearance in many aspects. A similar processing flow is also used in the multi-feature attention modules MFA1 and MFA2.

[0038] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0039] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An image dehazing model based on a multi-feature attention mechanism, characterized in that: Expanding on the U-Net model and integrating the multi-feature attention module, the following steps are included: S1.1, extract features from the input image through the initial feature extraction layer to obtain a residual block, and save the residual block and the feature extraction layer through a jump connection; S1.2, downsampling the obtained residual block through a convolution operation to obtain a downsampling layer, and then performing feature extraction on the downsampling layer to obtain a feature extraction layer, and saving the feature extraction layer through the jump connection; S1.3, extracting features from the feature extraction layer through a feature attention module, and integrating the residual blocks saved in the jump connection to obtain the deep residual blocks, and saving the residual blocks through the jump connection; S1.4, repeat steps S1.2 and S1.3 n times until the deepest feature extraction layer is generated; S1.5, extracting features from the deepest feature extraction layer through the feature attention module to obtain an upsampling layer; S1.6, extracting features from the upsampling layer to obtain a residual attention layer, integrating the feature extraction layer stored in the jump connection through a multi-feature attention module, and concatenating the integration result with the residual attention layer to generate the upsampling layer of the upper layer; S1.

7. Repeat step S1.6 n-1 times until the top upsampling layer is generated, extract features from the top upsampling layer to obtain the top residual attention layer, integrate the feature extraction layer saved in the jump connection through the multi-feature attention module, and concatenate the integration result with the top residual attention layer to obtain the feature output layer, and finally obtain the dehazed output image.

2. The image defogging model based on a multi-feature attention mechanism as claimed in claim 1, characterized in that: The integration process of the multi-feature attention module described in steps S1.6 and S1.7 is as follows: S2.1, performing pooling processing on the feature extraction layer stored in the skip connection and integrating it; S2.2, processing the integrated data through the pixel attention module to obtain intermediate data; S2.

3. Add the intermediate data to the deepest feature extraction layer element by element to obtain the processed upper residual attention layer.

3. The image defogging model based on a multi-feature attention mechanism as claimed in claim 2, characterized in that: The pooling process in step S2.1 is performed by a maximum pooling algorithm.

4. The image defogging model based on a multi-feature attention mechanism as claimed in claim 1, characterized in that: The feature attention module consists of a channel attention module, a pixel attention module and a skip connection.

5. The image defogging model based on a multi-feature attention mechanism as claimed in claim 4, characterized in that: The process of processing data through the channel attention module is: S3.

1. Perform global average pooling on the input data to extract the global information of the channel to form a dimension of 1×1×C The feature map of S3.2, compressing the feature map obtained in step S3.1 according to the ratio of the number of channels, and then restoring it to the original number of channels through the convolution operation, and finally using the Sigmoid activation function to map the generated value to the range of 0 to 1, and obtaining the attention weight corresponding to each channel; S3.

3. Multiply the data in step S3.1 and the attention weight obtained in step S3.2 element by element to obtain processed data.

6. The image defogging model based on a multi-feature attention mechanism according to any one of claims 2 and 4, characterized in that: The process of processing data through the pixel attention module is as follows: S4.1, the dimension is H×W×C The data is input into a convolutional layer with a ReLU linear rectification function to obtain a dimension of H×W×1 The feature map of S4.2, then after processing through a convolutional layer with a Sigmoid activation function, the attention weight of each pixel is obtained; S4.

3. Multiply the data in step S4.1 by the attention weight obtained in step S4.2 element by element to obtain processed data.

7. An image defogging method, characterized in that: The following steps are involved: S5.1, obtaining an image with fog by a camera device; S5.2, inputting the image into an image defogging model based on a multi-feature attention mechanism as described in any one of claims 1 to 6 for defogging; S5.

3. Obtain a dehazed image.

Images