A defogging method based on foggy traffic road images

Abstract

A dehazing method based on foggy traffic road images embeds wavelet transform into the DehazeFormer network. Through wavelet transform, information of different frequencies can be obtained, thereby reducing halo and blooming effects and improving image clarity. To address the low contrast problem of foggy traffic road images, selective kernel feature fusion is introduced in the encoder. By adaptively selecting and weighting features, the image contrast is enhanced and the ability to distinguish objects and roads in low-contrast foggy images is improved, thereby enhancing the image contrast. To address the blur and distortion problems of foggy traffic road images, the local residual structure introduces the CBMA attention mechanism and the content-guided attention mechanism CGA, which helps solve the blur and distortion problems. The network shows excellent performance in improving the contrast of foggy images, reducing the blur and distortion of foggy images, and also shows excellent performance in the detection task after dehazing.

Description

Technical Field

[0001] The present invention relates to the field of deep learning image processing, and in particular to a defogging method based on foggy traffic road images. Background Art

[0002] In terms of foggy road detection technology, processing foggy images is crucial. This technology can help monitor and identify road conditions, reduce the risk of traffic accidents in foggy weather, and improve road safety. Fog is usually caused by aerosol particles and water vapor, which scatter and absorb visible light, resulting in reduced visibility. In foggy weather, particles in the atmosphere absorb, reflect and scatter light, resulting in reduced contrast, loss of color information and lack of clarity in images captured by the camera. To solve this problem, foggy image processing technology is needed to improve the quality and readability of images, including defogging technology to reduce the impact of haze on images and improve image clarity and contrast for better identification of targets on the road.

[0003] Existing defogging algorithms are mainly divided into three categories: algorithms based on image enhancement technology, algorithms based on image restoration technology, and algorithms based on deep learning technology. Image enhancement methods mainly include global histogram equalization, Retinex, homomorphic filtering, etc. However, due to the large number of targets and complex traffic in foggy traffic scenes, the image enhancement defogging method has limited effect. Image restoration methods are based on the physical model of atmospheric degradation, but often require additional information or are affected by light and impurities. Defogging algorithms based on deep learning have gradually become a research trend, which can directly restore foggy images.

[0004] Deep learning dehazing algorithms are mainly divided into two categories. One is to use neural networks to predict the transmission map or atmospheric light value, and then solve the fog-free image based on the atmospheric scattering model. This type of algorithm is generally non-end-to-end; the other is to directly use neural networks to learn the mapping from foggy images to fog-free images. Most of them are end-to-end processes. Tang et al. proposed a non-end-to-end image dehazing algorithm based on machine learning. Li et al. first proposed "AOD-NET: An All-in-One Network for Dehazing and Beyond", which is an end-to-end dehazing model based on CNN network, which directly restores foggy images. Qin et al. proposed "FFA-net: Feature fusion attention network for single image dehazing", which combines channel attention mechanism and pixel attention mechanism to improve the flexibility of the network. However, these methods may still produce some artifacts, so further optimization is needed. Song et al. proposed "Vision transformers for single image dehazing" based on Swin Transformerd to improve the normalization layer, activation function and spatial information aggregation, making it more suitable for image dehazing.

[0005] The defogging algorithm in foggy road detection technology is an important part of improving image quality and readability. Its development trend is to combine deep learning technology, but it still needs further optimization to cope with complex traffic scenarios. Summary of the invention

[0006] The purpose of the present invention is to address the technical problem that in foggy days, particles in the atmosphere will cause the light propagation path to be extended, thereby reducing the image contrast; at the same time, when the light is reflected from the surface of the object to the camera device, scattering will occur. The above situation will cause the collected light to change, resulting in blurred detail information and color distortion in the image, which can easily lead to the impact of foggy weather on the quality of traffic road images. A defogging technology for foggy traffic road images is proposed.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A defogging method based on foggy traffic road images comprises the following steps:

[0009] Step 1: Input the fog image into the encoder of the network and output the feature maps of the three stages of the encoder;

[0010] Step 2: Send the output feature map of the third stage of the encoding part in step 1 to the intermediate feature layer module, and obtain an information-rich feature map after feature conversion at the intermediate feature layer;

[0011] Step 3: Input the feature maps output by the three stages of the encoder in step 1 into the corresponding jump connection layers respectively, and output three layers of enhanced texture detail information feature maps;

[0012] Step 4: Concatenate the feature map output by the intermediate feature layer in step 2 and the feature map output by the jump connection layer corresponding to the third stage of the encoder in step 3 in the channel dimension to obtain a feature map, and use point-by-point convolution operation to reduce the number of parameters of the concatenated feature map to meet the input requirements of the decoder, and then send the adjusted feature map to the corresponding decoder to obtain the feature map after the decoder; and so on, repeat step 4 until the feature map output by the decoder is obtained;

[0013] Step 5: Fuse the fog image in step 1 and the feature map output by the decoder in step 4 to obtain a clean feature map;

[0014] The foggy road image is dehazed through the above steps.

[0015] In step 1, the encoder is composed of three stages based on the DehazeFormer module, and the SKFF feature fusion module is added on this basis. The first stage consists of a Patch Embedding and a Transformer encoder, and the last two stages each consist of a Patch Merging and a Transformer encoder; specifically, the following steps are included:

[0016] The input fog image is input through the block embedding Patch Embedding in the first stage encoder. The block embedding Patch Embedding converts the input fog image into a series of patch representations, which helps to capture local information in the image. The output of the block embedding Patch Embedding in the first stage encoder is connected to the input of the Transformer encoder in the first stage encoder. The first selective kernel feature fusion SKFF operation is performed on the output feature map of the block embedding Patch Embedding in the first stage encoder and the output feature layer of the Transformer encoder in the first stage encoder to obtain the first stage output feature map F1 of the encoder;

[0017] The feature map F1 output by the first stage of the encoder is used as the input of the Patch Merging in the second stage encoder. The downsampling of the feature map is achieved by Patch Merging. The output of Patch Merging in the second stage encoder is connected to the input of the Transformer encoder in the second stage encoder. The second selective kernel feature fusion SKFF operation is performed on the output feature map of Patch Merging in the second stage encoder and the output feature layer of the Transformer encoder in the second stage encoder to obtain the input feature map F2 of the second stage encoder.

[0018] The second-stage encoder output feature map F2 is used as the input of the block merging Patch Merging in the third-stage encoder. The downsampling of the feature map is achieved through block merging Patch Merging. The output of the block merging Patch Merging in the third-stage encoder is connected to the input of the Transformer encoder in the third-stage encoder. The third selective kernel feature fusion SKFF operation is performed on the output feature map of the block merging Patch Merging in the third-stage encoder and the output feature layer of the Transformer encoder in the third-stage encoder to obtain the third-stage encoder input feature map F3.

[0019] The SKFF feature fusion module is implemented as follows:

[0020] (1) Receive the input of two parallel convolutional streams carrying information of different scales and integrate the multi-scale features L in the fog image: L = L1 + L2;

[0021] (2) Use global average pooling GAP on the feature map L in the spatial dimension to calculate the statistical information s of each channel;

[0022] (3) Apply a channel-wise downscaling convolution 1x1 to each channel’s statistics s to generate a compact feature representation z;

[0023] (4) The feature vector z is upgraded through two parallel channels of 1x1 convolution to obtain two feature descriptions v1 and v2;

[0024] (5) Use the softmax function to generate activation vectors s1 and s2 for feature description v1 and feature description v2;

[0025] (6) The activation vector s1 and the activation vector s2 are adaptively recalibrated to the multi-scale feature map L1 and the feature map L2 respectively. The formula is: U = s1·L1+s2·L2, where L1 refers to the feature map output by the Transformer, L2 refers to the feature map not output by the Transformer, s1 and s2 are both activation vectors, and U is the final feature map.

[0026] In step 2, the intermediate feature layer is composed of a detail enhancement module DS, a CBAM attention mechanism module, and a content-guided attention mechanism fusion module CGAFusion. The intermediate feature layer performs feature conversion on the feature map output by the third stage of the encoding part in step 1, which specifically includes the following steps:

[0027] The feature map F3 is used as the input of the first detail enhancement module DS. The output of the first detail enhancement module DS is connected to the input of the ReLu activation function. The feature map F3 and the output of the ReLu activation function are added to obtain the feature map after the first addition.

[0028] The feature map after the first addition is used as the input of the second detail enhancement module DS. The output of the second detail enhancement module DS is connected to the input of the CBAM attention mechanism module. The feature map F3 and the output of the CBAM attention mechanism module are added to obtain the feature map after the second addition.

[0029] The feature map F3 and the feature map after the second addition are subjected to the content-guided attention mechanism fusion module CGAFusion operation to obtain the feature map F4 output by the intermediate feature layer.

[0030] The detail enhancement module DS model in the intermediate feature layer is as follows:

[0031] The feature map F3 output by the third stage of the encoder is used as the input of the depthwise separable convolution DSConv and the ReLu activation function. The feature map F3 output by the third stage of the encoder and the output of the depthwise separable convolution DSConv and the ReLu activation function are added to obtain the feature map after the first addition.

[0032] The output of the feature map after the first addition is used as the input of the 3x3 convolution, and the output of the 3x3 convolution is used as the input of the content-guided attention module CGA. The output of the 3x3 convolution and the output of the content-guided attention CGA are multiplied element by element;

[0033] The feature map F3 output by the third stage of the encoder is added to the output of the element-by-element multiplication operation to obtain the final output.

[0034] The model of the content-guided attention module CGA in the detail enhancement module DS in the intermediate feature layer is as follows:

[0035] (1) Let the feature map be x∈R C×H×W And represented as input features, the input feature map is passed through the spatial attention mechanism and the channel attention mechanism to obtain the channel spatial information weight matrix W s and the channel related information matrix W c ,in, Among them, the output of the ReLU activation function is C 5x5 (·) represents a 5×5 convolutional layer, C 3x3 (·) represents a 3×3 convolutional layer, C 1x1 (·) represents a 1×1 convolutional layer, Represents a channel connection operation, where represents global average pooling across spatial dimensions, is the global average pooling across channel dimensions, is the feature of the global maximum pooling operation across the channel dimension;

[0036] (2) In order to make the model have better nonlinear expression ability and to preserve the spatial information of the fog image, the channel related information matrix W c And the channel space information weight matrix W s The information is fused and the rough fog image feature information W is obtained. cos ∈R C×H×W , where W cos =W s +W c ;

[0037] (3) In order to obtain fine fog image feature information, the fog image feature information Wcos and feature Figure X Each channel is rearranged in an alternating manner through the channel sorting operation to generate the final fog image feature information W = σ (GC 5×5 (GC 3×3 (CS([X,W cos ]))))), where σ represents the sigmoid activation function operation, GC 5×5 represents a group convolutional layer with a kernel size of 5×5, GC 3×3 represents a group convolutional layer with a kernel size of 3×3, CS([X, W cos ]) represents the channel sorting operation, [X, W cos ] indicates a channel connection operation.

[0038] In step 2, the attention mechanism module CBAM steps in the intermediate feature layer are as follows:

[0039] (1) For the input feature map, a channel attention module is first performed: the global maximum pooling MaxPool and global average pooling AvgPool operations are performed on each channel of the input feature map, and the feature vector after the global maximum pooling MaxPool and global average pooling AvgPool operations is input into a shared fully connected layer Shared MLP to obtain an attention weight vector. The obtained weight vector is added element by element and the Sigmoid activation function is applied to obtain the final attention weight. The final attention weight is multiplied by each channel of the input feature map to obtain the weighted channel feature map Mc;

[0040] (2) Perform spatial attention module on the channel attention map Mc obtained in the above step (1): use the channel attention map Mc as the input feature map of the spatial attention module, perform maximum pooling MaxPool and average pooling AvgPool operations along the channel dimension on the input feature map to obtain two feature maps with channel 1, concatenate these two feature maps with channel 1 along the channel dimension, use a 7x7 convolution to reduce the channel dimension of the concatenated feature map to obtain a feature map with channel 1, apply Sigmoid activation function to the feature map with channel 1 after dimension reduction to obtain spatial attention weight, multiply the obtained spatial attention weight with the input feature map Mc, that is, weight the features of each spatial position to obtain spatial attention Ms, Ms is the final generated feature.

[0041] In step 2, the content in the intermediate feature layer guides the attention mechanism fusion module CGAFusion algorithm as follows:

[0042] (1) Fuse the input low-level feature map and high-level feature map;

[0043] (2) Using the content-guided attention module CGA to calculate the spatial weights of the fused feature maps to modulate the features, thereby obtaining the weights of the low-level feature maps and the high-level feature maps as W and 1-W respectively;

[0044] (3) The calculated weights are weighted and summed for the low-level and high-level features. The formula is: F low ×W+F high ×(1-w);

[0045] (4) Add input features through skip connections to alleviate the gradient vanishing problem. The formula is: F low ×W+F high ×(1-w)+F low +F high ;

[0046] (5) The final fused features are projected through a 1×1 convolutional layer to obtain the final feature F fuse, the formula is: F fuse =C 1×1 (F low ×W+F high ×(1-w)+F low +F high );

[0047] Among them, C 1x1 (·) represents a 1×1 convolutional layer, F low is the low-level feature map, F high is the high-level feature map, W is the weight of the low-level feature map, and 1-W is the weight of the high-level feature map.

[0048] In step 3, the skip connection layer WDI is composed of a wavelet transform method and a Dehazeformer module, and the skip connection layer WDI performs texture feature processing on each order feature map of the encoder; specifically, the following steps are included:

[0049] The feature map F1 passes through the first jump connection layer WDI → the feature map F2 passes through the second jump connection layer WDI → the feature map F3 passes through the third jump connection layer WDI.

[0050] It is characterized in that the steps of skip connection layer WDI processing are as follows:

[0051] The feature map F1 is used as the wavelet input of the first stage of the skip connection layer. The output of the wavelet of the first stage of the skip connection layer is connected to the input of the Dehazeformer module of the first stage of the skip connection layer. The output of the Dehazeformer module of the first stage of the skip connection layer is connected to the input of the inverse wavelet transform of the first stage of the skip connection layer.

[0052] The feature map F2 is used as the wavelet input of the second stage of the skip connection layer. The output of the wavelet of the second stage of the skip connection layer is connected to the input of the Dehazeformer module of the second stage of the skip connection layer. The output of the Dehazeformer module of the second stage of the skip connection layer is connected to the input of the inverse wavelet transform of the second stage of the skip connection layer.

[0053] The feature map F3 serves as the wavelet input of the third stage of the skip connection layer, the output of the wavelet of the third stage of the skip connection layer is connected to the input of the Dehazeformer module of the third stage of the skip connection layer, and the output of the Dehazeformer module of the third stage of the skip connection layer is connected to the input of the inverse wavelet transform of the third stage of the skip connection layer.

[0054] In step 4, the following steps are specifically included:

[0055] The feature map F4 output by the intermediate feature layer and the output feature map of the third stage inverse wavelet transform of the jump connection layer are concatenated in the channel dimension and used as the input of the third stage point-by-point convolution. The output of the third stage point-by-point convolution is connected to the input of the third stage decoder.

[0056] The output of the third-stage decoder and the output feature map of the inverse wavelet transform of the second stage of the jump connection layer are concatenated in the channel dimension and used as the input of the second-stage point-by-point convolution. The output of the second-stage point-by-point convolution is connected to the input of the second-stage decoder.

[0057] The output feature map of the second-stage decoder and the inverse wavelet transform of the first stage of the jump connection layer are concatenated in the channel dimension and used as the input of the point-by-point convolution of the first stage. The output of the point-by-point convolution of the first stage is connected to the input of the first-stage decoder; the output of the first-stage decoder is the feature map output by the decoder.

[0058] Compared with the prior art, the present invention has the following technical effects:

[0059] 1) The present invention implements a DehazeFormer dehazing network based on wavelet transform. The wavelet transform processes image texture and details in the jump connection part. In order to make the network model pay more attention to texture and edges, an attention mechanism is added to the intermediate feature extraction layer, and an SKFF feature fusion module is added on the basis of Dehazeformer as the codec. By introducing the channel attention mechanism, feature maps of different scales are adjusted for fusion to obtain global and local features, and the details and semantic information in the image are better understood to construct a DehazeFormer dehazing network based on wavelet transform.

[0060] 2) The present invention embeds wavelet transform into the DehazeFormer network. Through wavelet transform, information of different frequencies can be obtained, thereby reducing halo and blooming effects and improving image clarity. To address the problem of low contrast in foggy traffic road images, selective kernel feature fusion is introduced in the encoder. By adaptively selecting and weighting features, the image contrast is enhanced and the ability to distinguish objects and roads in low-contrast foggy images is improved, thereby enhancing the image contrast. To address the blur and distortion problems of foggy traffic road images, the local residual structure introduces the CBMA attention mechanism and the content-guided attention mechanism CGA, which helps solve the blur and distortion problems. The network shows excellent performance in improving the contrast of foggy images, reducing the blur and distortion of foggy images, and also shows outstanding performance in the detection task after dehazing.

[0061] 3) The model of the present invention is applied to the public dataset cityscapes_foggy to defog foggy images. The experimental results show that its PSNR value reaches 27.56, SSIM value reaches 0.9558, and MSE reaches 168.85; in the dataset 4K-HAZE, its PSNR value reaches 24.22, SSIM value reaches 0.9025, and MSE reaches 367.17. In the synthetic dataset DKITTI (synthesized on KITTI), the PSNR value reaches 24.52, SSIM value reaches 0.826, and MSE reaches 283.48, which exceeds the currently popular defogging algorithms and demonstrates the excellent defogging effect of the model of the present invention. The method is used to detect the defogging images, and each indicator is also clearly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0063] Figure 1 It is the image defogging network diagram of the present invention;

[0064] Figure 2 It is the SKFF feature aggregation module;

[0065] Figure 3 The residual network structure of the intermediate feature layer of the present invention;

[0066] Figure 4 The DS module in the residual network structure of the intermediate feature layer of the present invention;

[0067] Figure 5 Guide the attention mechanism module for CGA content;

[0068] Figure 6 , Figure 7 , Figure 8 It is the CBAM attention mechanism module;

[0069] Fig. 9 It is the CGAFusion feature aggregation module;

[0070] Fig.10 It is the wavelet transform graph. DETAILED DESCRIPTION

[0071] A defogging method based on foggy traffic road images comprises the following steps:

[0072] Step 1: Input the fog image into the encoder of the network and output the feature maps of the three stages of the encoder;

[0073] Step 2: Send the output feature map of the third stage of the encoding part in step 1 to the intermediate feature layer module, and obtain an information-rich feature map after feature conversion at the intermediate feature layer;

[0074] Step 3: Input the feature maps output by the three stages of the encoder in step 1 into the corresponding jump connection layers respectively, and output three layers of enhanced texture detail information feature maps;

[0075] Step 4: Concatenate the feature map output by the intermediate feature layer in step 2 and the feature map output by the jump connection layer corresponding to the third stage of the encoder in step 3 in the channel dimension to obtain a feature map, and use point-by-point convolution operation to reduce the number of parameters of the concatenated feature map to meet the input requirements of the decoder, and then send the adjusted feature map to the corresponding decoder to obtain the feature map after the decoder; and so on, repeat step 4 until the feature map output by the decoder is obtained;

[0076] Step 5: Fuse the fog image in step 1 and the feature map output by the decoder in step 4 to obtain a clean feature map;

[0077] The foggy road image is dehazed through the above steps.

[0078] In step 1, the encoder is composed of three stages based on the DehazeFormer module, and the SKFF feature fusion module is added on this basis. The first stage consists of a Patch Embedding and a Transformer encoder, and the last two stages each consist of a Patch Merging and a Transformer encoder; specifically, the following steps are included:

[0079] The input fog image is input through the block embedding Patch Embedding in the first stage encoder. The block embedding Patch Embedding converts the input fog image into a series of patch representations, which helps to capture local information in the image. The output of the block embedding Patch Embedding in the first stage encoder is connected to the input of the Transformer encoder 2 in the first stage encoder. The first selective kernel feature fusion SKFF3 operation is performed on the output feature map of the block embedding Patch Embedding in the first stage encoder and the output feature layer of the Transformer encoder 2 in the first stage encoder to obtain the first stage output feature map F1 of the encoder;

[0080] The first-stage output feature map F1 of the encoder is used as the input of the block merging Patch Merging in the second-stage encoder. The downsampling of the feature map is achieved by block merging Patch Merging. The output of the block merging Patch Merging in the second-stage encoder is connected to the input of the Transformer encoder 5 in the second-stage encoder. The second selective kernel feature fusion SKFF6 operation is performed on the output feature map of the block merging Patch Merging in the second-stage encoder and the output feature layer of the Transformer encoder 5 in the second-stage encoder to obtain the second-stage input feature map F2 of the encoder;

[0081] The second-stage encoder output feature map F2 is used as the input of the block merging Patch Merging in the third-stage encoder. The downsampling of the feature map is achieved through block merging Patch Merging. The output of the block merging Patch Merging in the third-stage encoder is connected to the input of the Transformer encoder 8 in the third-stage encoder. The third selective kernel feature fusion SKFF9 operation is performed on the output feature map of the block merging Patch Merging in the third-stage encoder and the output feature layer of the Transformer encoder 8 in the third-stage encoder to obtain the third-stage encoder input feature map F3.

[0082] The SKFF feature fusion module is implemented as follows:

[0083] (1) Receive the input of two parallel convolutional streams carrying information of different scales and integrate the multi-scale features L in the fog image: L = L1 + L2;

[0084] (2) Use global average pooling GAP on the feature map L in the spatial dimension to calculate the statistical information s of each channel;

[0085] (3) Apply a channel-wise downscaling convolution 1x1 to each channel’s statistics s to generate a compact feature representation z;

[0086] (4) The feature vector z is upgraded through two parallel channels of 1x1 convolution to obtain two feature descriptions v1 and v2;

[0087] (5) Use the softmax function to generate activation vectors s1 and s2 for feature description v1 and feature description v2;

[0088] In step 2, the intermediate feature layer is composed of a detail enhancement module DS, a CBAM attention mechanism module, and a content-guided attention mechanism fusion module CGAFusion. The intermediate feature layer performs feature conversion on the feature map output by the third stage of the encoding part in step 1, which specifically includes the following steps:

[0089] The feature map F3 is used as the input of the first detail enhancement module DS. The output of the first detail enhancement module DS is connected to the input of the ReLu activation function. The feature map F3 and the output of the ReLu activation function are added to obtain the feature map after the first addition.

[0090] The feature map after the first addition is used as the input of the second detail enhancement module DS, the output of the second detail enhancement module DS is connected to the input of the CBAM attention mechanism module 13, and the feature map F3 and the output of the CBAM attention mechanism module 13 are added to obtain the feature map after the second addition;

[0091] The feature map F3 and the feature map after the second addition are subjected to the content-guided attention mechanism fusion module CGAFusion operation to obtain the feature map F4 output by the intermediate feature layer.

[0092] The detail enhancement module DS model in the intermediate feature layer is as follows:

[0093] The feature map F3 output by the third stage of the encoder is used as the input of the depthwise separable convolution DSConv and the ReLu activation function. The feature map F3 output by the third stage of the encoder and the output of the depthwise separable convolution DSConv and the ReLu activation function are added to obtain the feature map after the first addition.

[0094] The output of the feature map after the first addition is used as the input of the 3x3 convolution, and the output of the 3x3 convolution is used as the input of the content-guided attention module CGA. The output of the 3x3 convolution and the output of the content-guided attention CGA are multiplied element by element;

[0095] The feature map F3 output by the third stage of the encoder is added to the output of the element-by-element multiplication operation to obtain the final output.

[0096] The model of the content-guided attention module CGA in the detail enhancement module DS in the intermediate feature layer is as follows:

[0097] (1) Let the feature map be x∈R C×H×W And represented as input features, the input feature map is passed through the spatial attention mechanism and the channel attention mechanism to obtain the channel spatial information weight matrix W s and the channel related information matrix W c ,in, Among them, the output of the ReLU activation function is C 5x5 (·) represents a 5×5 convolutional layer, C 3x3 (·) represents a 3×3 convolutional layer, C 1x1 (·) represents a 1×1 convolutional layer, Represents a channel connection operation, where represents global average pooling across spatial dimensions, is the global average pooling across channel dimensions, is the feature of the global maximum pooling operation across the channel dimension;

[0098] (2) In order to make the model have better nonlinear expression ability and to preserve the spatial information of the fog image, the channel related information matrix W c And the channel space information weight matrix W s The information is fused and the rough fog image feature information W is obtained. cos ∈R C×H×W , where W cos =W s +W c ;

[0099] (3) In order to obtain fine fog image feature information, the fog image feature information Wcos and feature Figure X Each channel is rearranged in an alternating manner through the channel sorting operation to generate the final fog image feature information W = σ (GC 5×5 (GC 3×3 (CS([X,W cos ]))))), where σ represents the sigmoid activation function operation, GC 5×5 represents a group convolutional layer with a kernel size of 5×5, GC 3×3 represents a group convolutional layer with a kernel size of 3×3, CS([X, W cos ]) represents the channel sorting operation, [X, W cos ] indicates a channel connection operation.

[0100] In step 2, the attention mechanism module CBAM steps in the intermediate feature layer are as follows:

[0101] (1) For the input feature map, a channel attention module is first performed: the global maximum pooling MaxPool and global average pooling AvgPool operations are performed on each channel of the input feature map, and the feature vector after the global maximum pooling MaxPool and global average pooling AvgPool operations is input into a shared fully connected layer Shared MLP to obtain an attention weight vector. The obtained weight vector is added element by element and the Sigmoid activation function is applied to obtain the final attention weight. The final attention weight is multiplied by each channel of the input feature map to obtain the weighted channel feature map Mc;

[0102] (2) Perform spatial attention module on the channel attention map Mc obtained in (1): Use the channel attention map Mc as the input feature map of the spatial attention module. Perform maximum pooling MaxPool and average pooling AvgPool operations along the channel dimension on the input feature map to obtain two feature maps with channel 1. These two feature maps with channel 1 are concatenated along the channel dimension. A 7x7 convolution is used to reduce the channel dimension of the concatenated feature map to obtain a feature map with channel 1. The Sigmoid activation function is applied to the feature map with channel 1 after dimension reduction to obtain the spatial attention weight. The obtained spatial attention weight is multiplied with the input feature map Mc, that is, the feature of each spatial position is weighted to obtain the spatial attention Ms. Ms is the final generated feature.

[0103] In step 2, the content in the intermediate feature layer guides the attention mechanism fusion module CGAFusion algorithm as follows:

[0104] (1) Fuse the input low-level feature map and high-level feature map;

[0105] (2) Using the content-guided attention module CGA to calculate the spatial weights of the fused feature maps to modulate the features, thereby obtaining the weights of the low-level feature maps and the high-level feature maps as W and 1-W respectively;

[0106] (3) The calculated weights are weighted and summed for the low-level and high-level features. The formula is: F low ×W+F high ×(1-w);

[0107] (4) Add input features through skip connections to alleviate the gradient vanishing problem. The formula is: F low ×W+F high ×(1-w)+F low +F high ;

[0108] (5) The final fused features are projected through a 1×1 convolutional layer to obtain the final feature F fuse , the formula is: Ffuse =C 1×1 (F low ×W+F high ×(1-w)+F low +F high );

[0109] Among them, C 1x1 (·) represents a 1×1 convolutional layer, F low is the low-level feature map, F high is the high-level feature map, W is the weight of the low-level feature map, and 1-W is the weight of the high-level feature map.

[0110] In step 3, the skip connection layer WDI is composed of a wavelet transform method and a Dehazeformer module, and the skip connection layer WDI performs texture feature processing on each order feature map of the encoder; specifically, the following steps are included:

[0111] The feature map F1 passes through the first jump connection layer WDI → the feature map F2 passes through the second jump connection layer WDI → the feature map F3 passes through the third jump connection layer WDI.

[0112] The steps for WDI processing at the skip connection layer are as follows:

[0113] The feature map F1 is used as the wavelet input of the first stage of the skip connection layer. The output of the wavelet of the first stage of the skip connection layer is connected to the input of the Dehazeformer module of the first stage of the skip connection layer. The output of the Dehazeformer module of the first stage of the skip connection layer is connected to the input of the inverse wavelet transform of the first stage of the skip connection layer.

[0114] The feature map F2 is used as the wavelet input of the second stage of the skip connection layer. The output of the wavelet of the second stage of the skip connection layer is connected to the input of the Dehazeformer module of the second stage of the skip connection layer. The output of the Dehazeformer module of the second stage of the skip connection layer is connected to the input of the inverse wavelet transform of the second stage of the skip connection layer.

[0115] The feature map F3 serves as the wavelet input of the third stage of the skip connection layer, the output of the wavelet of the third stage of the skip connection layer is connected to the input of the Dehazeformer module of the third stage of the skip connection layer, and the output of the Dehazeformer module of the third stage of the skip connection layer is connected to the input of the inverse wavelet transform of the third stage of the skip connection layer.

[0116] In step 4, the following steps are specifically included:

[0117] The feature map F4 output by the intermediate feature layer and the output feature map of the third stage inverse wavelet transform of the jump connection layer are concatenated in the channel dimension and used as the input of the third stage point-by-point convolution 18. The output of the third stage point-by-point convolution 18 is connected to the input of the third stage decoder 19.

[0118] The output of the third stage decoder 19 and the output feature map of the inverse wavelet transform of the second stage of the jump connection layer are concatenated in the channel dimension and used as the input of the second stage point-by-point convolution 20. The output of the second stage point-by-point convolution 20 is connected to the input of the second stage decoder 21.

[0119] The output of the second-stage decoder 21 and the output feature map of the inverse wavelet transform of the first stage of the jump connection layer are spliced ​​in the channel dimension and used as the input of the first-stage point-by-point convolution 22. The output of the first-stage point-by-point convolution 22 is connected to the input of the first-stage decoder 23; the output of the first-stage decoder 23 is the feature map output by the decoder.

[0120] In order to illustrate the dehazing effect of the model constructed by the present invention, the comparative experiment compares the model of the present invention with the network models of dark channel, AOD-Net, Wavelt-Net, FFA-Net, EPDN and DehazeFormer image dehazing qualitatively and quantitatively. The experimental results show that the present invention surpasses the currently popular dehazing algorithms and demonstrates the excellent dehazing effect of the model of the present invention. The method is used to test the dehazed images, and each indicator is also clearly improved.

[0121] Table 1 Comparison of test results in Foggy_Cityscapes dataset

[0122]

[0123] Table 2 Comparison of test results in 4K-HAZE dataset

[0124]

[0125] Table 3 Comparison of test results in the DKITTI dataset

[0126]

[0127] In order to show the results of the detection after dehazing, the dark channel, AOD-Net, Wavelt-Net, FFA-Net, EPDN and DehazeFormer image dehazing are compared on the public dataset DKITTI (KITTI synthetic fog dataset). The results are shown in the following table:

[0128] Table 2 Detection indicators without defogging algorithm

[0129]

[0130] The detection map value of the method after defogging is 72.3%, which is 19.8% higher than the original fog image, 25.0% for the dark channel, 26.8% for the AOD-Net algorithm, 42.1% and 0.3% for the Wavelet-Net and FFA-Net algorithms, 22.2% for the EPDN algorithm, and 1.8% for the DehazeFormer algorithm. Therefore, the defogging algorithm of the present invention shows excellent performance in improving the contrast of fog images, reducing the blur and distortion of fog images, and reducing the loss of details, which provides help for subsequent road traffic detection and recognition.

[0131] To this end, the present invention implements a target detection algorithm based on foggy roads. In the defogging network, a U-shaped network architecture is used, and the network model is strengthened by a codec structure composed of Dehazeformer and SKFF, an intermediate feature layer, and a jump connection layer. The model of the present invention is applied to defog fog images in the public data set cityscapes_foggy, where there are 2767 training sets, 345 validation sets, and 345 test sets. The loss function uses MSELoss, the learning rate is 0.0004, the batchsize is set to 16, and the number of training rounds is 50. After defogging, detection is performed, and different defogging results are detected using the yolo series. Experiments show that both the defogging index and the detection index have been improved to a certain extent, proving that the method is feasible.

Claims

1. A defogging method based on foggy traffic road images, characterized in that: The following steps are involved: Step 1: Input the fog image into the encoder of the network and output the feature maps of the three stages of the encoder; Step 2: Send the output feature map of the third stage of the encoding part in step 1 to the intermediate feature layer module, and obtain an information-rich feature map after feature conversion at the intermediate feature layer; Step 3: Input the feature maps output by the three stages of the encoder in step 1 into the corresponding jump connection layers respectively, and output three layers of enhanced texture detail information feature maps; Step 4: Concatenate the feature map output by the intermediate feature layer in step 2 and the feature map output by the jump connection layer corresponding to the third stage of the encoder in step 3 in the channel dimension to obtain a feature map, and use point-by-point convolution operation to reduce the number of parameters of the concatenated feature map to meet the input requirements of the decoder, and then send the adjusted feature map to the corresponding decoder to obtain the feature map after the decoder; And so on, repeat step 4 until the feature map output by the decoder is obtained; Step 5: Fuse the fog image in step 1 and the feature map output by the decoder in step 4 to obtain a clean feature map; Through the above steps, foggy road images are defogged; In step 2, the intermediate feature layer includes a detail enhancement module DS, a CBAM attention mechanism module, and a content-guided attention mechanism fusion module CGAFusion; In step 3, the skip connection layer WDI consists of a wavelet transform module, a Dehazeformer module, and an inverse wavelet transform module; In step 1, the encoder is composed of three stages based on the DehazeFormer module, and the SKFF feature fusion module is added on this basis. The first stage consists of a Patch Embedding and a Transformer encoder, and the last two stages each consist of a Patch Merging and a Transformer encoder; specifically, the following steps are included: The input fog image is input to the block embedding Patch Embedding (1) in the first stage encoder. The block embedding Patch Embedding (1) converts the input fog image into a series of patch representations, which helps to capture local information in the image. The output of the block embedding Patch Embedding (1) in the first stage encoder is connected to the input of the Transformer encoder (2) in the first stage encoder. The first selective kernel feature fusion SKFF (3) operation is performed on the output feature map of the block embedding Patch Embedding (1) in the first stage encoder and the output feature layer of the Transformer encoder (2) in the first stage encoder to obtain the first stage output feature map F1 of the encoder. The feature map F1 output by the first stage of the encoder is used as the input of the Patch Merging (4) in the second stage encoder. The downsampling of the feature map is achieved by the Patch Merging (4) in the second stage encoder. The output of the Patch Merging (4) in the second stage encoder is connected to the input of the Transformer encoder (5) in the second stage encoder. The second selective kernel feature fusion SKFF (6) operation is performed on the output feature map of the Patch Merging (4) in the second stage encoder and the output feature layer of the Transformer encoder (5) in the second stage encoder to obtain the input feature map F2 of the second stage encoder. The second stage encoder output feature map F2 is used as the input of the Patch Merging (7) in the third stage encoder. The downsampling of the feature map is achieved through the Patch Merging (7) in the third stage encoder. The output of the Patch Merging (7) in the third stage encoder is connected to the input of the Transformer encoder (8) in the third stage encoder. The third selective kernel feature fusion SKFF (9) operation is performed on the output feature map of the Patch Merging (7) in the third stage encoder and the output feature layer of the Transformer encoder (8) in the third stage encoder to obtain the third stage encoder input feature map F3.

2. The method according to claim 1, characterized in that: The SKFF feature fusion module is implemented as follows: (1) Receive the input of two parallel convolutional streams carrying information of different scales and integrate the multi-scale features L in the fog image: L = L1 + L2; (2) Use global average pooling GAP on the feature map L in the spatial dimension to calculate the statistical information s of each channel; (3) Apply a channel downscaling convolution 1x1 to each channel’s statistics s to generate a compact feature representation z; (4) The feature vector z is upgraded through two parallel channels of 1x1 convolution to obtain feature description v1 and feature description v2; (5) Use the softmax function to generate activation vectors s1 and s2 for feature description v1 and feature description v2; (6) The activation vector s1 and the activation vector s2 are adaptively recalibrated to the multi-scale feature map L1 and the feature map L2 respectively, the formula is: U = s1 L1+s2 L2, where L1 refers to the feature map output by Transformer, L2 refers to the feature map not output by Transformer, s1 and s2 are both activation vectors, and U is the final feature map.

3. The method according to claim 1, characterized in that In step 2, the intermediate feature layer includes a detail enhancement module DS, a CBAM attention mechanism module, and a content-guided attention mechanism fusion module CGAFusion. The intermediate feature layer performs feature conversion on the feature map output by the third stage of the encoding part in step 1, which specifically includes the following steps: The feature map F3 is used as the input of the first detail enhancement module DS (10). The output of the first detail enhancement module DS (10) is connected to the input of the ReLu activation function (11). The feature map F3 and the output of the ReLu activation function (11) are added to obtain the feature map after the first addition. The feature map after the first addition is used as the input of the second detail enhancement module DS (12). The output of the second detail enhancement module DS (12) is connected to the input of the CBAM attention mechanism module (13). The feature map F3 and the output of the CBAM attention mechanism module (13) are added to obtain the feature map after the second addition. The feature map F3 and the feature map after the second addition are subjected to the content-guided attention mechanism fusion module CGAFusion (14) to obtain the feature map F4 output by the intermediate feature layer.

4. The method according to claim 3, characterized in that The detail enhancement module DS model in the intermediate feature layer is as follows: The feature map F3 output by the third stage of the encoder is used as the input of the depthwise separable convolution DSConv and the ReLu activation function. The feature map F3 output by the third stage of the encoder and the output of the depthwise separable convolution DSConv and the ReLu activation function are added to obtain the feature map after the first addition. The output of the feature map after the first addition is used as the input of the convolution, and the output of the convolution is used as the input of the content-guided attention module CGA. The output of the convolution and the output of the content-guided attention CGA are multiplied element by element; The feature map F3 output by the third stage of the encoder is added to the output of the element-by-element multiplication operation to obtain the final output.

5. The method according to claim 4, characterized in that The model of the content-guided attention module CGA in the detail enhancement module DS in the intermediate feature layer is as follows: (1) Let the feature map And represented as input features, the input feature map is passed through the spatial attention mechanism and the channel attention mechanism to obtain the channel spatial information weight matrix W s and the channel related information matrix W c ,in, , , where the output of the ReLU activation function is , C 5x5 (·) represents a 5×5 convolutional layer, C 3x3 (·) represents a 3×3 convolutional layer, C 1x1 (·) represents a 1×1 convolutional layer, Represents a channel connection operation, where , is the global average pooling across channel dimensions, is the feature of the global maximum pooling operation across the channel dimension; (2) In order to make the model have better nonlinear expression ability and to preserve the spatial information of the fog image, the channel related information matrix W c And the channel space information weight matrix W s Information is fused and rough fog image feature information is obtained ,in, ; (3) In order to obtain fine fog image feature information, the fog image feature information Wcos and each channel of the feature map X are rearranged in an alternating manner through the channel sorting operation to generate the final fog image feature information ,in, Represents the sigmoid activation function operation, GC 5×5 represents a group convolutional layer with a kernel size of 5×5, GC 3×3 represents a group convolutional layer with a 3×3 kernel size, represents the channel sorting operation, Represents a channel connection operation.

6. The method according to claim 3, characterized in that: In step 2, the attention mechanism module CBAM steps in the intermediate feature layer are as follows: (1) For the input feature map, the channel attention module is first performed: the global maximum pooling MaxPool and global average pooling AvgPool operations are performed on each channel of the input feature map, and the feature vector after the global maximum pooling MaxPool and global average pooling AvgPool operations is input into a shared fully connected layer Shared MLP to obtain the attention weight vector. The obtained weight vector is added element by element and the Sigmoid activation function is applied to obtain the final attention weight. The final attention weight is multiplied by each channel of the input feature map to obtain the weighted channel feature map Mc; (2) Perform spatial attention module on the channel attention map Mc obtained by step (1): use the channel attention map Mc as the input feature map of the spatial attention module, perform maximum pooling MaxPool and average pooling AvgPool operations along the channel dimension on the input feature map to obtain two feature maps, concatenate the two feature maps along the channel dimension, use convolution to reduce the channel dimension of the concatenated feature map to obtain a feature map, apply the Sigmoid activation function to the reduced feature map to obtain the spatial attention weight, and multiply the obtained spatial attention weight with the input feature map Mc, that is, weight the features of each spatial position to obtain the spatial attention Ms, which is the final generated feature.

7. The method according to claim 3, characterized in that In step 2, the content in the intermediate feature layer guides the attention mechanism fusion module CGAFusion algorithm as follows: (1) Fuse the input low-level feature map and high-level feature map; (2) The content-guided attention module CGA is used to calculate the spatial weights of the fused feature maps to modulate the features, thereby obtaining the weights of the low-level feature maps and the high-level feature maps, respectively, W and 1-W; (3) The calculated weights are weighted and summed for the low-level and high-level features. The formula is: ; (4) Add input features through skip connections to alleviate the gradient vanishing problem. The formula is: ; (5) The final fused features are projected through a 1 × 1 convolutional layer to obtain the final features , the formula is: ; Among them, C 1x1 (·) represents a 1×1 convolutional layer, is the low-level feature map, is the high-level feature map, W is the weight of the low-level feature map, and 1-W is the weight of the high-level feature map.

8. The method according to claim 1, characterized in that In step 3, the skip connection layer WDI is composed of a wavelet transform module, a Dehazeformer module and an inverse wavelet transform module, and the skip connection layer WDI performs texture feature processing on each order feature map of the encoder; specifically, the following steps are included: The feature map F1 passes through the first skip connection layer WDI (15) → the feature map F2 passes through the second skip connection layer WDI (16) → the feature map F3 passes through the third skip connection layer WDI (17).

9. The method according to claim 8, characterized in that The steps for WDI processing at the skip connection layer are as follows: The feature map F1 is used as the wavelet input of the first stage of the skip connection layer. The output of the wavelet of the first stage of the skip connection layer is connected to the input of the Dehazeformer module of the first stage of the skip connection layer. The output of the Dehazeformer module of the first stage of the skip connection layer is connected to the input of the inverse wavelet transform of the first stage of the skip connection layer. The feature map F2 is used as the wavelet input of the second stage of the skip connection layer. The output of the wavelet of the second stage of the skip connection layer is connected to the input of the Dehazeformer module of the second stage of the skip connection layer. The output of the Dehazeformer module of the second stage of the skip connection layer is connected to the input of the inverse wavelet transform of the second stage of the skip connection layer. The feature map F3 serves as the wavelet input of the third stage of the skip connection layer, the output of the wavelet of the third stage of the skip connection layer is connected to the input of the Dehazeformer module of the third stage of the skip connection layer, and the output of the Dehazeformer module of the third stage of the skip connection layer is connected to the input of the inverse wavelet transform of the third stage of the skip connection layer.

Images