Non-uniform image defogging method based on visual Transform multi-scale collaborative self-attention

Abstract

The invention discloses a non-uniform image defogging method based on visual Transform multi-scale collaborative self-attention. The method comprises the following steps: S1, screening to obtain a real non-uniform haze image data set; s2, processing images in the real non-uniform haze image data set by using a data set enhancement strategy; s3, constructing a multi-scale channel-pixel joint attention module; s4, a visual Transform multi-scale collaborative self-attention module is constructed; s5, a multi-scale channel-pixel joint attention module and a visual Transform multi-scale collaborative self-attention module are fused and crossed to construct a multi-scale collaborative self-attention non-uniform image defogging model; and S6, taking an image in the enhanced real non-uniform haze image data set as an input image, inputting the input image into the multi-scale collaborative self-attention non-uniform image defogging model, and outputting to obtain a defogged image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a non-uniform image defogging method based on visual Transformer multi-scale collaborative self-attention. Background Art

[0002] With the acceleration of industrialization and the improvement of urbanization, the frequency of haze weather is increasing in many areas, especially in large cities. Under haze weather conditions, the tiny particles suspended in the atmosphere will significantly reduce the transparency of the air, resulting in serious quality degradation of images taken by outdoor imaging equipment, such as blurred images, reduced contrast and color distortion. This not only affects people's intuitive perception of image content, but also has an adverse impact on image-based information interpretation and decision-making, especially in the fields of traffic monitoring, space exploration and marine navigation, which require extremely high accuracy. It also leads to difficulties in subsequent tasks such as object recognition and image processing, and seriously affects the subsequent research and development of computer systems to obtain image information. For example, during autonomous driving, the presence of haze will reduce visibility, blur the images captured by the on-board system camera, affect the system's visual perception ability, and may cause the autonomous driving system to be unable to accurately identify road signs, pedestrians, other vehicles passing through the road, etc., thereby causing the risk of traffic accidents. During space flight, the presence of haze may interfere with radio communications and satellite navigation signals, affect the navigation and communication systems of the aircraft, and may cause the aircraft to be unable to maintain normal contact with the ground control center, thereby causing the risk of flight accidents. When traveling in the field of navigation, the presence of haze on the sea surface makes it impossible for navigation instruments to locate in time, thus causing marine traffic accidents such as ship stranding and collision. In this context, image haze removal and restoration of clear images have become an urgent and important research direction in the field of image processing. Summary of the invention

[0003] In view of the above problems, the present invention aims to provide a non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention, which can reduce the impact of haze and improve image clarity.

[0004] In order to achieve the above object, the technical solution of the present invention is: The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention includes the following steps: S1. Screen and obtain a real non-uniform haze image dataset for model training to improve the model defogging performance; S2, using the dataset enhancement strategy to process the images in the real non-uniform haze image dataset to enhance the generalization ability of the real non-uniform haze image dataset; S3, constructing a multi-scale channel-pixel joint attention module, which extracts haze features from input features according to different importance levels; S4. Construct a visual Transformer multi-scale collaborative self-attention module, which extracts coarse and fine granular detail features of the image from the input features; S5, a non-uniform image dehazing model with multi-scale collaborative self-attention is constructed by integrating the multi-scale channel-pixel joint attention module and the visual Transformer multi-scale collaborative self-attention module; S6. The enhanced image in the real non-uniform haze image dataset is input as the input image into the multi-scale collaborative self-attention non-uniform image dehazing model, and the dehazed image is output.

[0005] Furthermore, in the step S1, when screening the real non-uniform haze image dataset, the images in the real non-uniform haze image dataset are cut into 64 parts on average to increase the data volume of the images.

[0006] Furthermore, in step S2, the data set enhancement strategy is: random horizontal cropping and flipping at different angles are performed on the images in the real non-uniform haze image data set as data expansion.

[0007] Furthermore, in step S3, the multi-scale channel-pixel joint attention module is composed of a multi-scale channel module and a pixel attention module; and its construction process includes the following steps: S31, after the input features are subjected to GAP, the haze features containing channels are obtained, the haze features are subjected to two 1×1 convolutions, and the ReLU activation function is added to further refine the haze features; S32, sending the refined haze features to the sigmoid function to convert them into weight coefficient values ​​between 0 and 1, and dividing them into four groups of channel weight coefficients, which are respectively multiplied with the multi-scale features extracted by the input features through the convolution kernels 3×3, 5×5, 7×7, and 9×9, and then adding different weight coefficients to different channels, so that different channels have haze detail features of different importance; S33, splicing the obtained haze detail features together to obtain fused output features; S34, passing the output features to the pixel attention module, sequentially undergoing 1×1 convolution, ReLU, and 1×1 convolution to reduce the number of channels and obtain the extracted features of a single channel; S35. Send the extracted features to the sigmoid function to obtain single-channel features containing different pixel coefficients, and multiply them with the input features to add different weight values ​​to the haze features at different positions to obtain haze detail features containing important information.

[0008] Furthermore, in step S4, the visual Transformer multi-scale collaborative self-attention module is stacked by a residual block consisting of several identical and parallel self-attention layers, feedforward network layers, and layer normalization; its construction process includes the following steps: S41, the input features are transformed through the linear transformation layer to obtain Q, K and V; in the same parallel self-attention layer, Q is kept unchanged and Q is divided into Q1 and Q2; S42, in the multi-head self-attention layer, 8 heads are used to extract input features respectively, and they are divided into: the features extracted by the first 4 heads are downsampled 4 times to obtain K1 and V1, and the features extracted by the last 4 heads are downsampled 2 times to obtain K2 and V2, which are responsible for extracting coarse and fine granular features of different scales respectively; among them, V1 is transposed convolution to obtain V12, and then added to V2 to obtain V2 after the fusion of different coarse and fine features; similarly, V2 is convolved 3×3 to obtain V21, and then added to V1 to obtain the feature V1 after the fusion of different coarse and fine features; S43, the feature V1 after fusion of different coarse and fine features is calculated with Q1 and K1 to obtain the attention weight, and it is multiplied with the feature V2 after fusion of different coarse and fine features and the attention weight calculated with Q2 and K2 to obtain the final fused feature , ; S44, the final fusion features extracted by the multi-head self-attention layer and Splicing and parallel calculation to obtain the final self-attention feature ; For the multi-head self-attention layer, K and V are downsampled to different sizes, and the specific implementation is shown in the following formula: ; in, Indicates the K and V downsampling multiples. represents the input features, , , Respectively represent The query matrix, key matrix, and value matrix of each head; Indicates Aggregation layer of multi-scale tokens; , , Indicates The parameter matrix of the linear projection of each head; yes The local enhancement component after depth-wise transposed convolution, yes Local enhancement component after depthwise convolution.

[0009] Furthermore, in step S5, constructing a multi-scale collaborative self-attention non-uniform image defogging model includes the following steps: S51, extracting local features from the input image through 3×3 convolution, passing the extracted local features to the multi-scale channel-pixel joint attention module to further extract haze detail features, and assigning different coefficient weights to the extracted haze detail features from different angles, retaining important detail information, ensuring that the subsequent network model fully learns the local detail features of the haze image at different scales, and making the network model pay attention to the feature information of different haze areas; S52, performing a 2-fold downsampling operation on the extracted haze detail features, so that the multi-head attention mechanism in the visual Transformer multi-scale collaborative self-attention module can extract rich semantic features in parallel, and fuse the extracted different haze detail features to obtain output features; S53, performing a 2-fold upsampling operation on the output features, sending the features extracted by the 2-fold upsampling operation to a multi-scale channel-pixel joint attention module to extract important features, and performing a 2-fold upsampling operation on the important features again, so that the output image size is restored to the original size; S54. Between up and down sampling, 1×1 convolution short connections are used between features of the same size to fuse the underlying fine-grained features with high-level semantic information to improve the overall image understanding ability and make up for the missing underlying feature information.

[0010] Compared with the prior art, the present invention has the following advantages and positive effects: The present invention improves the defogging performance and generalization ability of the image defogging algorithm by averaging and screening the real non-uniform haze image data set. At the same time, a non-uniform image defogging model based on visual Transformer multi-scale collaborative self-attention is proposed, and a clear haze-free image is reconstructed through the non-uniform image defogging model with multi-scale collaborative self-attention. The non-uniform image dehazing model adds a multi-scale channel-pixel joint attention module. The input image can add different weights to channels of different scale features after passing through the multi-scale channel module, so that different channels can focus on different haze areas, which can not only suppress less useful features but also retain important feature details; then it is passed to the pixel attention module, and different weights are added to the haze concentration, so that the joint attention weight can focus on haze areas at different positions, and then different weights are given to haze areas at different positions, so that the dehazing model can obtain better texture details according to the degree of attention; at the same time, the non-uniform image dehazing model adds a visual Transformer parallel aggregation self-attention mechanism through the visual Transformer multi-scale collaborative self-attention module, which uses the Q, K, V mechanism of multi-head self-attention to extract haze feature information, which can ensure that V of different scales can collaboratively extract detail features of different scales and perform parallel fusion, and finally obtain the fused detail features. This operation can extract global coarse and fine granularity features, reduce feature loss, and then reconstruct a clear haze-free image.

[0011] The image defogging technology in the present invention can improve the contrast and clarity of the image, making the targets in the image easier to detect and identify, and effectively solves the problems of blurred details and low contrast in the images taken under severe weather conditions such as foggy days. It can provide more accurate real-time image analysis and decision support for fields such as autonomous driving, space exploration and marine navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 It is a schematic diagram of the framework of the non-uniform image dehazing model; Figure 2 Schematic diagram of the framework of the multi-scale channel-pixel joint attention module; Figure 3 Schematic diagram of the framework of the multi-scale collaborative self-attention module of the visual Transformer. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work, any modifications, equivalent substitutions, improvements, etc., should be included in the protection scope of the present invention.

[0015] This embodiment discloses a non-uniform image defogging method based on visual Transformer multi-scale collaborative self-attention, the method comprising the following steps: 1. Perform average cropping and screening on the data in the real non-uniform image haze dataset.

[0016] 2. Use the dataset enhancement strategy to process the images in the dataset. Random horizontal flipping and cropping are used to expand the real non-uniform image haze dataset to enhance the generalization ability of the dataset.

[0017] 3. Construct a non-uniform image dehazing model based on visual Transformer multi-scale collaborative self-attention and input the image data in the real non-uniform image haze dataset into it, and output the dehazed image.

[0018] like Figure 1 As shown in Figure 2, the construction process of the non-uniform image dehazing model is as follows: Firstly, the input image is subjected to ordinary 3×3 convolution to extract the relatively low-level local features in the image, and the extracted local features are passed to the multi-scale channel-pixel joint attention module to further extract the haze detail features. Different coefficient weights are assigned to the extracted features of different haze areas from different angles to retain important detail information, ensuring that the subsequent network fully learns the local detail features of the haze image at different scales, so that the dehazing network pays attention to the feature information of different haze areas.

[0019] Secondly, in high-level feature extraction, the extracted haze features are first downsampled by 2 times to increase the receptive field, so that the multi-head attention mechanism in the visual Transformer multi-scale collaborative self-attention module can extract rich semantic features in parallel, and fuse the different detail features extracted to reduce feature loss; that is, the Q in the multi-head self-attention mechanism is divided into Q1 and Q2, and by changing the lengths of K and V, K1 and V1, K2 and V2 are obtained, and different Vs are subjected to different convolution operations, V2 and V12 after 4×4 transposed convolution operation are added to obtain a new V2, and V1 and V21 after 3×3 convolution operation are added to obtain a new V1. This ensures that more detailed detail features extracted from V of different scales can be fused, and more reliable detail features can be fully obtained.

[0020] Then, the output features are upsampled by 2 times, and the upsampled features are sent to the multi-scale channel-pixel joint attention module to extract important features, and then upsampled by 2 times again to restore the image size to its original size.

[0021] Finally, between up and down sampling, ordinary 1×1 convolution short connections are used between features of the same size to fuse the underlying features with the high-level semantic features to make up for the missing underlying features.

[0022] The non-uniform image dehazing model is composed of a fusion of a multi-scale channel-pixel joint attention module and a visual Transformer multi-scale collaborative self-attention module. The multi-scale channel-pixel joint attention module can extract haze features from input features according to their importance; it consists of a multi-scale channel module and a pixel attention module; like Figure 2 As shown in Figure 2, the construction process of the multi-scale channel-pixel joint attention module is as follows: After the input haze feature passes through GAP, the haze feature with channel is obtained. After two 1×1 convolutions, the ReLU activation function is added to increase the nonlinearity of the network and further refine the extracted features. The refined features are sent to the sigmoid function to convert them into weight coefficient values ​​between 0 and 1, and the channel weight coefficients are divided into four groups, which are multiplied with the multi-scale features extracted by the input features through the convolution kernels 3×3, 5×5, 7×7, and 9×9, respectively, and then different weights are added to different channels, so that different channels have haze detail features of different importance. The obtained detail features are spliced ​​together to obtain the fused features, and the output features are consistent with the input feature channels. The output features are passed to the pixel attention module, and after 1×1 convolution, ReLU, and 1×1 convolution, the number of channels is reduced to obtain a single channel. The extracted features are sent to the sigmoid function to obtain single-channel features with different pixel coefficients, which are multiplied with the input features to add different weight values ​​to the extracted haze features at different positions to obtain haze detail features containing important information.

[0023] The visual Transformer multi-scale collaborative self-attention module is composed of a stack of residual blocks consisting of several identical and parallel self-attention layers, feedforward network layers, and layer normalization. The present invention designs a visual Transformer multi-scale collaborative self-attention network, and learns the diversion self-attention feature information of different granularities in parallel in its multi-head self-attention layer. Specifically, each self-attention head of the multi-head self-attention changes K and V of different sizes, and through different convolution operations, the V of different self-attention heads are added and fused, so that the coarse and fine granularity features extracted by different heads can be fused, reducing feature loss and enhancing feature information reuse.

[0024] like Figure 3 As shown in Figure 1, the working principle of the visual Transformer multi-scale collaborative self-attention module is: Input Features After the linear transformation layer, Q, K and V are obtained; in order to capture information of different granularities, in the same head-parallel self-attention layer, Q is kept unchanged and divided into Q1 and Q2. By changing the length of K and V, K1 and V1, K2 and V2 are obtained, and different Vs are subjected to different convolution operations. V2 is added to V12 after the 4×4 transposed convolution operation to obtain a new V2, and V1 is added to V21 after the 3×3 convolution operation to obtain a new V1. This ensures that the multi-scale detail features extracted collaboratively by Vs of different scales are fused to obtain the fused detail features.

[0025] The specific construction process is as follows: First, input features After the linear transformation layer, we get Q, K and V; Secondly, in the multi-head self-attention layer, 8 heads are used to extract input features respectively, and the features extracted by the first 4 heads are downsampled 4 times to obtain K1 and V1, and the features extracted by the last 4 heads are downsampled 2 times to obtain K2 and V2, which are responsible for extracting coarse and fine granular features of different scales respectively; among them, in the fusion of coarse and fine granular features, V1 is transposed convolution to obtain V12, and then added to V2 to obtain V2 after the fusion of different coarse and fine features; the same value V2 is convoluted 3×3 to obtain V21, and then added to V1 to obtain the fused feature V1. And V1, V2 and the corresponding Q1, K1 calculated to obtain the attention weights and Q2, K2 calculated to obtain the final fused features , .

[0026] Finally, the features extracted from all heads are and Concatenate and parallelize the final self-attention features For different self-attention heads, K and V are downsampled to different sizes as shown in the following formula.

[0027]

[0028] in, Indicates the K, V downsampling multiple ( ), represents the input features, , , Respectively represent The query matrix, key matrix, and value matrix of the self-attention head. Different convolution kernels and different step sizes are used to achieve changes in different scales. Indicates Aggregation layer of multi-scale tokens. , , Indicates The parameter matrix of the linear projection of the head. yes The local enhancement component after depth-wise transposed convolution, yes The local enhancement component after deep convolution. V of different sizes are superimposed on each other through convolution operations to fuse coarse and fine granular features. According to the calculation formula between Q, K, and V Calculate detailed features of decentralized parallel aggregation.

[0029] The present invention improves the defogging performance and generalization ability of the image defogging algorithm by averaging and screening the real non-uniform haze image data set. At the same time, a non-uniform image defogging model based on visual Transformer multi-scale collaborative self-attention is proposed, and a clear haze-free image is reconstructed through the non-uniform image defogging model with multi-scale collaborative self-attention. The non-uniform image dehazing model adds a multi-scale channel-pixel joint attention module. The input image can add different weights to channels of different scale features after passing through the multi-scale channel module, so that different channels can focus on different haze areas, which can not only suppress less useful features but also retain important feature details; then it is passed to the pixel attention module, and different weights are added to the haze concentration, so that the joint attention weight can focus on haze areas at different positions, and then different weights are given to haze areas at different positions, so that the dehazing model can obtain better texture details according to the degree of attention; at the same time, the non-uniform image dehazing model adds a visual Transformer parallel aggregation self-attention mechanism through the visual Transformer multi-scale collaborative self-attention module, which uses the Q, K, V mechanism of multi-head self-attention to extract haze feature information, which can ensure that V of different scales can collaboratively extract detail features of different scales and perform parallel fusion, and finally obtain the fused detail features. This operation can extract global coarse and fine granularity features, reduce feature loss, and then reconstruct a clear haze-free image.

[0030] The image defogging technology in the present invention can improve the contrast and clarity of the image, making the targets in the image easier to detect and identify, and effectively solves the problems of blurred details and low contrast in the images taken under severe weather conditions such as foggy days. It can provide more accurate real-time image analysis and decision support for fields such as autonomous driving, space exploration and marine navigation.

Claims

1. A non-uniform image dehazing method based on multi-scale collaborative self-attention of visual Transformer, characterized by: The following steps are involved: S1. Screen and obtain a real non-uniform haze image dataset for model training to improve the model defogging performance; S2, using the dataset enhancement strategy to process the images in the real non-uniform haze image dataset to enhance the generalization ability of the real non-uniform haze image dataset; S3, constructing a multi-scale channel-pixel joint attention module, which extracts haze features from input features according to different importance levels; S4. Construct a visual Transformer multi-scale collaborative self-attention module, which extracts coarse and fine granular detail features of the image from the input features; S5, a non-uniform image dehazing model with multi-scale collaborative self-attention is constructed by integrating the multi-scale channel-pixel joint attention module and the visual Transformer multi-scale collaborative self-attention module; S6. The enhanced image in the real non-uniform haze image dataset is input as the input image into the multi-scale collaborative self-attention non-uniform image dehazing model, and the dehazed image is output.

2. The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention as claimed in claim 1, characterized in that: In the step S1, when screening the real non-uniform haze image data set, the images in the real non-uniform haze image data set are cut into 64 parts on average to increase the data volume of the images.

3. The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention as claimed in claim 2, characterized in that: In step S2, the data set enhancement strategy is: random horizontal cropping and flipping at different angles are performed on the images in the real non-uniform haze image data set as data expansion.

4. The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention as claimed in claim 3, characterized in that: In step S3, the multi-scale channel-pixel joint attention module is composed of a multi-scale channel module and a pixel attention module; The construction process includes the following steps: S31, after the input features are subjected to GAP, the haze features containing channels are obtained, the haze features are subjected to two 1×1 convolutions, and the ReLU activation function is added to further refine the haze features; S32, sending the refined haze features to the sigmoid function to convert them into weight coefficient values ​​between 0 and 1, and dividing them into four groups of channel weight coefficients, which are respectively multiplied with the multi-scale features extracted by the input features through the convolution kernels 3×3, 5×5, 7×7, and 9×9, and then adding different weight coefficients to different channels, so that different channels have haze detail features of different importance; S33, splicing the obtained haze detail features together to obtain fused output features; S34, passing the output features to the pixel attention module, sequentially undergoing 1×1 convolution, ReLU, and 1×1 convolution to reduce the number of channels and obtain the extracted features of a single channel; S35. Send the extracted features to the sigmoid function to obtain single-channel features containing different pixel coefficients, and multiply them with the input features to add different weight values ​​to the haze features at different positions to obtain haze detail features containing important information.

5. The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention as claimed in claim 4, characterized in that: In step S4, the visual Transformer multi-scale collaborative self-attention module is composed of a stack of residual blocks consisting of several identical and parallel self-attention layers, feedforward network layers, and layer normalization; The construction process includes the following steps: S41, the input features are transformed through the linear transformation layer to obtain Q, K and V; in the same parallel self-attention layer, Q is kept unchanged and Q is divided into Q1 and Q2; S42, in the multi-head self-attention layer, 8 heads are used to extract input features respectively, and they are divided into: the features extracted by the first 4 heads are downsampled 4 times to obtain K1 and V1, and the features extracted by the last 4 heads are downsampled 2 times to obtain K2 and V2, which are responsible for extracting coarse and fine granular features of different scales respectively; among them, V1 is transposed convolution to obtain V12, and then added to V2 to obtain V2 after the fusion of different coarse and fine features; similarly, V2 is convolved 3×3 to obtain V21, and then added to V1 to obtain the feature V1 after the fusion of different coarse and fine features; S43, the feature V1 after fusion of different coarse and fine features is calculated with Q1 and K1 to obtain the attention weight, and it is multiplied with the feature V2 after fusion of different coarse and fine features and the attention weight calculated with Q2 and K2 to obtain the final fused feature , ; S44, the final fusion features extracted by the multi-head self-attention layer and Splicing and parallel calculation to obtain the final self-attention feature ; For the multi-head self-attention layer, K and V are downsampled to different sizes, and the specific implementation is shown in the following formula: ； in, Indicates the K and V downsampling multiples. represents the input features, , , Respectively represent The query matrix, key matrix, and value matrix of each head; Indicates Aggregation layer of multi-scale tokens; , , Indicates The parameter matrix of the linear projection of each head; yes The local enhancement component after depth-wise transposed convolution, yes Local enhancement component after depthwise convolution.

6. The non-uniform image dehazing method based on visual Transformer multi-scale collaborative self-attention as claimed in claim 5, characterized in that: In step S5, constructing a multi-scale collaborative self-attention non-uniform image defogging model includes the following steps: S51, extracting local features from the input image through 3×3 convolution, passing the extracted local features to the multi-scale channel-pixel joint attention module to further extract haze detail features, and assigning different coefficient weights to the extracted haze detail features from different angles, retaining important detail information, ensuring that the subsequent network model fully learns the local detail features of the haze image at different scales, and making the network model pay attention to the feature information of different haze areas; S52, performing a 2-fold downsampling operation on the extracted haze detail features, so that the multi-head attention mechanism in the visual Transformer multi-scale collaborative self-attention module can extract rich semantic features in parallel, and fuse the extracted different haze detail features to obtain output features; S53, performing a 2-fold upsampling operation on the output features, sending the features extracted by the 2-fold upsampling operation to a multi-scale channel-pixel joint attention module to extract important features, and performing a 2-fold upsampling operation on the important features again, so that the output image size is restored to the original size; S54. Between up and down sampling, 1×1 convolution short connections are used between features of the same size to fuse the underlying fine-grained features with high-level semantic information to improve the overall image understanding ability and make up for the missing underlying feature information.

Images