Image defogging method based on dense feature extraction and multi-scale attention

Abstract

The invention discloses an image defogging network based on dense feature extraction and multi-scale attention, belongs to the field of image defogging, and aims at improving the quality of video images in severe weather such as haze and improving the application effect of the video images. According to the image defogging network based on dense feature extraction and multi-scale attention, a U-Net coding and decoding framework is adopted, a dense feature extraction module and a multi-scale attention mechanism are introduced, and the defogging capability of the network is enhanced. According to the designed feature encoder, the receptive field is expanded by using technologies such as separation sharing convolution and cavity convolution, and the feature extraction efficiency is improved; and the designed decoder realizes high-quality image restoration by combining jump connection and a feature fusion strategy. The method can be widely applied to the application fields of anti-unmanned aerial vehicles, intelligent transportation, automatic driving and the like, and video images are enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image defogging, and more particularly relates to an image defogging method based on dense feature extraction and multi-scale attention. BACKGROUND

[0002] With the digital intelligence of society, images have become an indispensable information dissemination approach. Under adverse weather, image quality is affected by haze, which can cause serious degradation and bring challenges to subsequent image processing and analysis applications.

[0003] Image defogging technology is an effective image quality enhancement processing method. Based on the formation mechanism of foggy images, the technology establishes a foggy image degradation model and an image enhancement processing method to realize the clarification of foggy images and achieve the purpose of improving image quality. At present, image defogging methods are mainly divided into three categories: image enhancement-based defogging algorithm, physical model-based defogging method and convolution network-based defogging method. The image enhancement-based defogging method mainly realizes the purpose of defogging by using contrast enhancement or filtering processing. The physical model-based defogging method simulates the degradation process of foggy images by establishing an atmospheric scattering model, and realizes image defogging processing through anti-degradation processing. The convolution network-based defogging method automatically learns the feature representation of foggy images by designing a convolution neural network structure, and can realize more effective defogging processing. However, the existing convolution neural network-based image defogging method still has problems such as high computational complexity, insufficient detail recovery, poor adaptability to multi-scale fog, etc., which affects the application effect of subsequent image processing and analysis tasks.

[0004] To solve the above problems, the application provides an image defogging network based on dense feature extraction and multi-scale attention to enhance the defogging effect. The method can effectively recover the key details of the scene that are degraded by fog, improve the quality of the defogged image, and meet the application requirements of subsequent image processing and analysis tasks. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the application provides an image defogging network based on dense feature extraction and multi-scale attention, which aims to effectively remove foggy city traffic images and enhance the details and textures of the images.

[0006] To achieve the above object, the application provides an image defogging network based on dense feature extraction and multi-scale attention, which adopts a 2-layer U-Net encoding-decoding structure and comprises an encoding end, a decoding end and an encoding-decoding end. The basic edge texture feature extraction unit adopts a common convolution module CB to process the input fog-containing image I to obtain a basic edge texture feature map of the image I , wherein b is a channel number, b = 1, …, K, and K is the total number of channels.

[0007] The dense attention feature encoding unit adopts two cascaded dense attention feature encoding modules DAE to encode the feature map in the encoding end to obtain the encoded feature . The multi-scale attention residual feature extraction unit adopts three cascaded multi-scale attention residual blocks MSA to process the feature map to obtain a feature map fused with multi-scale information . The multi-scale feature weighted fusion processing unit adopts two feature weighted fusion modules CWF with the same structure in the decoding end, adopts a channel-level multi-scale feature weighted fusion processing method, and performs weighted fusion on features at different levels, so that the feature fusion block can adaptively calculate the importance weight of each feature according to the channel information of the input feature, thereby realizing fine feature weighted fusion; wherein the feature weighted fusion module CWF at the bottom layer of the decoding end fuses the dense encoded feature and the multi-scale attention residual feature , the feature weighted fusion block CWF at the top layer of the decoding end performs weighted fusion on the output feature of the dense attention encoding block at the top layer of the encoding end and the output feature of the dense attention decoding block at the bottom layer of the decoding end, and obtains the fused feature . The dense attention feature decoding unit adopts two dense attention decoding blocks DAD with the same structure in the decoding end, and gradually restores the spatial resolution of the feature map through two times of decoding; wherein the output of the dense attention decoding block at the top layer of the decoding end is the feature .

[0008] Overall, compared with the prior art, the above technical scheme conceived by the application has the following beneficial effects: 1. The image defogging network based on dense feature extraction and multi-scale attention provided by the present application adopts a U-Net encoding-decoding framework, dynamically adjusts the importance of the features of the encoder and the decoder through an adaptive fusion block, establishes an accurate foggy image feature representation model, and improves the image defogging effect.

[0009] 2. The present application designs a dense feature extraction layer, which integrates separate shared convolution, dilated convolution and deformable convolution, expands the receptive field through dilated convolution to capture global information, improves the perception of geometric structure information and edge detail information in the image through deformable convolution, and reduces the model parameter quantity and enhances the information interaction between channels through separate shared convolution. The designed dense feature extraction module can improve the image defogging capability of the defogging network.

[0010] 3. The present application introduces a multi-scale attention residual block in the designed defogging network, which integrates ordinary convolution, separate shared convolution, dilated convolution technology and hybrid attention mechanism, enhances the extraction capability of the defogging network for multi-scale features in complex regions of the image, and can guide the network to focus on feature extraction in key regions. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The structure diagram of the image defogging network based on dense feature extraction and multi-scale attention provided by the present application.

[0012] Figure 2 The structure diagram of the ordinary convolution block designed by the present application.

[0013] Figure 3 The structure diagram of the dense feature extraction layer designed by the present application.

[0014] Figure 4 The structure diagram of the multi-scale attention residual block designed by the present application.

[0015] Figure 5 The structure diagram of the multi-scale feature weighted fusion block designed by the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0017] REFERENCE Figures 1-5An image defogging network based on dense feature extraction and multi-scale attention, the image defogging network network adopts a 2-layer U-Net encoding and decoding structure, including an encoding end, a decoding end and an encoding and decoding end; the encoding end includes one ordinary convolution module CB and two cascaded dense attention feature encoding modules DAE, the decoding end contains two multi-scale feature weighted fusion modules and two dense attention feature decoding modules, and the bottom layer connection of the encoding and decoding end is composed of three cascaded multi-scale attention residual blocks, and the image defogging network comprises: Basic edge texture feature extraction unit: an ordinary convolution module CB is used in the encoding end to process the input foggy image I, and the basic edge texture feature map of the image I is obtained , b is a channel number, b=1,…,K, and K is the total number of channels.

[0018] Dense attention feature encoding unit: two cascaded dense attention feature encoding modules DAE are used in the encoding end to encode the feature map in the encoding end, and the encoded feature is obtained. Multi-scale attention residual feature extraction: three cascaded multi-scale attention residual blocks MSA are used to process the feature map , and the feature map fused with multi-scale information is obtained . Multi-scale feature weighted fusion processing unit: two feature weighted fusion modules CWF with the same structure are used in the decoding end, and a channel-level multi-scale feature weighted fusion processing method is used to weight and fuse different levels of features. The feature fusion block can adaptively calculate the importance weight of each feature according to the channel information of the input feature, so as to realize fine feature weighted fusion; wherein the feature weighted fusion module CWF at the bottom layer of the decoding end fuses the dense encoded feature and the multi-scale attention residual feature , and the feature weighted fusion block CWF at the top layer of the decoding end weights and fuses the output feature of the dense attention encoding block at the top layer of the encoding end and the output feature of the dense attention decoding block at the bottom layer of the decoding end , and obtains the fused feature . Dense attention feature decoding unit: two dense attention decoding blocks DAD with the same structure are used in the decoding end, and the spatial resolution of the feature map is gradually restored through two decodings; wherein the output feature of the dense attention decoding block at the top layer of the decoding end .

[0019] In the basic edge texture feature extraction unit, the ordinary convolution block CB is composed of an ordinary convolution layer , a batch normalization layer BN and an activation function layer ReLU, and the feature The extraction process is as follows: .

[0020] In the dense attention feature encoding unit, the dense attention feature encoding module DAE is composed of a dense feature extraction layer DFE, a hybrid attention module CBAM layer and a down-sampling layer.

[0021] The dense feature extraction layer DFE includes a normal convolution block, a dilated convolution block, a deformable convolution block and a feature merging fusion block.

[0022] The normal convolution block in the dense feature extraction layer DFE is composed of a normal convolution layer , a batch normalization layer BN and an activation function layer ReLU, and the input of the normal convolution block is a feature map , and the processing process is as follows: ; wherein is the input feature map of the normal convolution block, and the output feature map after processing of the normal convolution block.

[0023] The dilated convolution block in the dense feature extraction layer DFE is composed of a split-shared convolution layer , a dilated convolution layer , a batch normalization layer and an activation function layer ReLU, which prevents the generation of artifacts in the dehazing process, enhances the visual effect, and expands the receptive field through dilated convolution to capture global information. The input of the dilated convolution block is a feature map and a feature map , and the processing process is as follows: ; wherein is the output feature map of the dilated convolution block.

[0024] The deformable convolution block in the dense feature extraction layer DFE is composed of a deformable convolution layer , a batch normalization layer BN and an activation function layer ReLU, which can adaptively adjust the position of each convolution sampling point by introducing an offset prediction, and enhance the modeling ability of the network for complex structures and fuzzy edge regions. The input of the deformable convolution block is a feature map , and , and the processing process is as follows: ; wherein is the output feature map of the deformable convolution block.

[0025] The feature fusion block in the dense feature extraction layer DFE is composed of a channel splicing layer Concat, a convolution layer Conv-1 and a feature addition layer, and the feature map is obtained after processing by the feature merging fusion block The processing process is as follows: ; Wherein represents the convolution layer Conv-1.

[0026] Therefore, the feature extraction process of the dense feature extraction layer can be simplified as: ); The mixed attention module in the dense attention encoding module adopts the classical CBAM (Convolutional Block Attention Module), and the perception ability of the model is improved through the mixed attention mechanism, so that the network can dynamically adjust the feature weight according to the distribution of the fog, and the processing process can be represented as: ); The down-sampling layer in the dense attention encoding block DAE includes an ordinary convolution block CB and a maximum pooling layer The resolution of the input feature map is reduced to 1 / 2 of the original, and the processing process is as follows: ; Therefore, the processing process of the dense attention encoding block (DAE) can be simplified as: ); In the encoding end described in the application, the feature map processed by the ordinary convolution block is further processed by the above-mentioned dense attention encoding block (DAE) in cascade, and the encoding feature of the encoding end can be obtained, and the processing process can be represented as: ); In the multi-scale attention residual feature extraction unit, three multi-scale attention residual blocks MSA with the same structure are cascaded for feature extraction.

[0027] For each multi-scale attention residual block MSA, let the feature map be , First, multi-scale feature extraction is performed by the first three-branch feature extraction module, then the multi-scale features are merged by the concat layer to obtain the feature map ; and then the feature map The input is input to a second three-branch feature extraction module with the same structure for multi-scale feature extraction, and the processed features are sequentially input to a concat layer, a convolution layer and a CBAM layer for processing to obtain a feature map .

[0028] The three-branch feature extraction module is composed of three convolutional blocks with different sizes, and the first branch and the second branch have similar structures and are composed of a normal convolution kernel and an activation layer ReLU, wherein the convolution kernel size of the first branch is , and the convolution kernel size of the second branch is ; For the first three-branch feature extraction module, the processing procedures of the first branch and the second branch are respectively: ; ; Among them, and are normal convolution layers of the first branch and the second branch; The third branch is composed of a split shared convolution , a dilated convolution and an activation layer ReLU, and the processing procedure of the third branch is: ; Then, the processing results of the three-branch feature extraction module are combined through a concat module to obtain a feature : ; The subsequent processing procedure is: ; ; ; ; .

[0029] As shown above, the processing procedure of the multi-scale attention residual block MSA can be simplified as: ; The network of the present application adopts three multi-scale attention residual blocks MSA for cascade, and processes the encoding features of the dense attention encoding block to obtain a feature​​​​ The processing process can be represented as: = MSA(MSA(MSA( ))); In the multi-scale feature weighted fusion processing unit, in order to realize more effective fusion between the encoder and the decoder features, the application designs a channel-level feature weighted fusion module CWF, which can adaptively calculate the importance weight of each feature according to the channel information of the input feature, so as to realize fine feature weighted fusion. The feature weighted fusion module CWF is composed of a global pooling layer GAP, a 1x1 convolution layer and a Sigmoid activation function layer; For the feature weighted fusion module CWF of the top layer of the decoding end, the two input features are respectively and , the feature dimensions are CxHxW, the input features are first processed by the global pooling layer GAP to obtain the pooled features and , and the processing process is: ; ; Among them and have dimensions of Cx1x1; Then, the channel level is mapped through the 1x1 convolution layer, and the feature channel weight is obtained through the Sigmoid function, that is: ; ; Among them and have sizes of Cx1x1, is a Sigmoid function, is a convolution kernel; Finally, the obtained feature channel weight is used to weight and fuse the input features to obtain the fused features : .

[0030] The processing process of the feature weighted fusion module CWF can be simplified as: ; In the dense attention feature decoding processing unit, the dense attention decoding block DAD is composed of an up-sampling layer , a CBAM layer and a dense feature extraction layer The dense feature extraction layer in the composition adopts the same structure as the dense feature extraction layer in the dense attention encoding block, and the up-sampling layer adopts a deconvolution operation For the dense attention decoding block of the decoding end top layer, the processing process is as follows: ; The dense attention decoding block DAD of the top layer is output.

[0031] On the basis of the above network, the application provides an image defogging network training process method based on dense feature extraction and multi-scale attention, comprising the following steps: S1, a pair of fog-containing images and clear images are prepared, and the images are randomly rotated by 90, 180 and 270 degrees and horizontally flipped to form a training data set; S2, setting the loss function of network training, the loss function designed by the application is a weighted fusion of mean absolute error loss (L1 loss) and contrast loss (L2 loss): ; ; Wherein I is the fog-containing image, K is the defogged image, and J is the clear image (reference image); D(p, q) is the L1 distance between features p and q; , i=1, 2,...5 is the ith contrast feature extracted from the image X using a certain feature extraction network, the application uses a VGG-19 pre-trained model as the feature extraction network, and selects the 1st, 3rd, 5th, 9th and 13th layers of the VGG-19 network as the 5 contrast features for measuring the contrast loss, and the corresponding coefficients and ; S3, setting the network training parameters, setting the iteration step number T as steps, selecting the Adam optimizer, and The parameters take the default values 0.9 and 0.999 respectively; setting the initial learning rate , using the cosine annealing strategy to adjust the learning rate, and the learning rate of the tth step is calculated as follows: ; S4, randomly extract a pair of images (fog-containing images I and clear images J of the same scene) from the training data set as the input of the network. The image defogging network based on dense feature extraction and multi-scale attention is used to defog the fog-containing image, and the defogged image K is obtained. The loss function set in S2 is used as a constraint, and the network parameters are adjusted;​​​ S5, according to the learning rate adjustment strategy and the training iteration number of S3, repeating the processing procedure of S4, completing the image defogging network training based on dense feature extraction and multi-scale attention, and obtaining the trained image defogging network model based on dense feature extraction and multi-scale attention.

[0032] The above is easily understood by those skilled in the art, and the above is only the preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image dehazing network based on dense feature extraction and multi-scale attention, characterized in that, The image dehazing network adopts a two-layer U-Net encoding / decoding structure, including an encoder, a decoder, and an encoder-decoder interface. The encoder includes a regular convolutional module (CB) and two cascaded dense attention feature encoding modules (DAE). The decoder includes two multi-scale feature weighted fusion modules and two dense attention feature decoding modules. The bottom-level connections of the encoder-decoder interface consist of three cascaded multi-scale attention residual blocks. The image dehazing network includes the following units: Basic edge texture feature extraction unit: At the encoding end, a regular convolutional module CB is used to process the input hazy image I to obtain the basic edge texture feature map of image I. b is the channel number, b=1,…,K, and K is the total number of channels; Dense Attention Feature Encoding Unit: Two cascaded Dense Attention Feature Encoding (DAE) modules are used at the encoding end to process the feature map. Perform feature encoding to obtain the encoded features. ; Multi-scale attention residual feature extraction unit: Employs three cascaded multi-scale attention residual blocks (MSA) for feature map extraction. The process is performed to obtain a feature map that integrates multi-scale information. ; Multi-scale feature weighted fusion processing unit: Two structurally identical feature weighted fusion modules (CWF) are used at the decoding end. A channel-level multi-scale feature weighted fusion processing method is employed to weight and fuse features at different levels. The feature fusion block can adaptively calculate the importance weight of each feature based on the channel information of the input features, thereby achieving fine-grained feature weighted fusion. Specifically, the CWF at the bottom layer of the decoding end handles densely encoded features. and multi-scale attention residual features The feature weighted fusion block (CWF) at the top layer of the decoder combines the output features of the dense attention coding block at the top layer of the encoder with the output features of the dense attention decoding block at the bottom layer of the decoder. Perform weighted fusion to obtain the fused features. ; Dense Attention Feature Decoding Unit: Two structurally identical dense attention decoding blocks (DADs) are used at the decoding end to gradually recover the spatial resolution of the feature map through two decoding steps; the output feature of the top-level dense attention decoding block at the decoding end... .

2. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 1, characterized in that, In the basic edge texture feature extraction unit, the ordinary convolutional block CB consists of an ordinary convolutional layer. It consists of a batch normalization layer (BN) and an activation function layer (ReLU), and features The extraction process is as follows: 。 3. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 1, characterized in that, In the dense attention feature encoding unit, the dense attention feature encoding module DAE consists of a dense feature extraction layer DFE, a mixed attention module CBAM layer, and a downsampling layer.

4. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 3, characterized in that, The dense feature extraction layer (DFE) includes ordinary convolutional blocks, dilated convolutional blocks, deformable convolutional blocks, and feature merging and fusion blocks. The ordinary convolutional block in the dense feature extraction layer (DFE) consists of a single ordinary convolutional layer. The convolutional block consists of a batch normalization (BN) layer and a ReLU activation function layer. The input is a feature map The processing procedure is as follows: ； in The input feature map is a regular convolutional block. This is the output feature map after processing a regular convolutional block; The dilated convolutional blocks in the dense feature extraction layer (DFE) are separated by shared convolutional layers. Hollow convolution layer Batch normalization layer It consists of ReLU activation function layers, with the input of the dilated convolutional block being the feature map. and feature map The processing procedure is as follows: ； in The output feature map of the dilated convolution block; The deformable convolutional blocks in the dense feature extraction layer (DFE) are composed of deformable convolutional layers. The system consists of a batch normalization (BN) layer and a ReLU activation function layer, with deformable convolutional blocks as inputs to feature maps. , and The processing procedure is as follows: ； in The output feature map of the deformable convolutional block; The feature fusion block in the Dense Feature Extraction (DFE) layer consists of a channel splicing layer (Concat), a convolutional layer (Conv-1), and a feature addition layer. The feature map is obtained after processing by the feature merging and fusion block. The processing procedure is as follows: ； in This represents the convolutional layer Conv-1.

5. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 4, characterized in that, The downsampling layers in the Dense Attention Encoding Block (DAE) include ordinary convolutional blocks (CBs) and max pooling layers. Input feature map The resolution is reduced to half of its original value, and the processing procedure is as follows: ； in, ), representing the output of the Mixed Attention Module (CBAM) layer.

6. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 1, characterized in that, In the multi-scale attention residual feature extraction unit, feature extraction is performed by cascading three structurally identical multi-scale attention residual blocks (MSA). For each multi-scale attention residual block (MSA), let the feature map be denoted as . , First, multi-scale feature extraction is performed using the first three-branch feature extraction module. Then, the multi-scale features are merged using the concat layer to obtain the feature map. Then the feature map The input is fed into a second three-branch feature extraction module with the same structure for multi-scale feature extraction, and the processed features are then passed sequentially through a concat layer and a convolutional layer. The feature map is obtained by processing the CBAM layer. Finally, the feature map and feature map The results are then fused to obtain the final processing result of multi-scale attention residual feature extraction. .

7. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 6, characterized in that, The three-branch feature extraction module consists of three convolutional blocks of different scales connected together. The first and second branches both consist of ordinary convolutional kernels and ReLU activation layers. The kernel size of the first branch is [missing information]. The kernel size of the second branch is ; For the first three-branch feature extraction module, the processing procedures for the first and second branches are as follows: ； ； in, and These are ordinary convolutional layers for the first and second branches; The third branch consists of separate shared convolutions. Diffuse convolution The third branch, consisting of the ReLU activation layer, proceeds as follows: ； Then, the processing results of the three-branch feature extraction module are combined using the concat module to obtain the features. : ； The subsequent processing procedure is as follows: ； ； ； ； 。 8. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 6, characterized in that, The feature weighted fusion module CWF consists of a global pooling layer GAP, a 1×1 convolutional layer, and a sigmoid activation function layer. For the feature weighted fusion module CWF at the top layer of the decoder, let its two input features be respectively and The feature dimensions are all C×H×W. The input features are first processed by a global pooling layer (GAP) to obtain the pooled features. and The processing procedure is as follows: ； ； in and The dimension is C×1×1; Then, channel-level mapping is performed using a 1×1 convolutional layer, and the feature channel weights are obtained using the Sigmoid function, i.e.: ； ； in and The sizes are C×1×1, For the Sigmoid function, For convolution kernel; Finally, the obtained feature channel weights are used to perform weighted fusion of the input features to obtain the fused features. : 。 9. The image dehazing network based on dense feature extraction and multi-scale attention according to claim 1, characterized in that, In the dense attention feature decoding processing unit, the dense attention decoding block (DAD) is generated by the upsampling layer. CBAM layer and dense feature extraction layer The composition includes a dense feature extraction layer with the same structure as the dense feature extraction layer in the dense attention coding block, and an upsampling layer using deconvolution operations. For the dense attention decoding block at the top layer of the decoding end, the processing procedure is as follows: ； This indicates the output of the top-level dense attention decoding block (DAD).

Images