Image defogging method based on context information aggregation and fusion of feature attention

Abstract

The application discloses an image defogging method based on context information aggregation and fusion feature attention, and comprises the following steps: for a foggy image, shallow features of the image are extracted through an encoder; context information is aggregated through an information fusion subnetwork according to the shallow features; and the image is restored through a decoder to obtain a defogged image. The method based on deep learning uses smooth dilated convolution to replace traditional dilated convolution to reduce grid artifacts, and through the fusion feature attention mechanism, channel attention and pixel-level attention are weighted, so that the problem that the defogged image is estimated inaccurately due to non-uniform scene haze and non-uniform channel haze can be solved. In addition, a structure of an autoencoder is adopted, context information aggregation network is used to fuse features at different levels, and a unique single-image defogging method is formed.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular to an image dehazing method based on contextual information aggregation and feature attention fusion. Background Technology

[0002] Shooting outdoors in foggy weather often results in reduced visibility, decreased contrast, blurred surfaces, and color distortion in videos and images due to the fog. This not only severely impacts visual quality but also causes a significant loss of image information. Most current computer vision algorithms are designed for scenarios with normal illumination and no fog interference, while reliable vision systems need to account for visual degradation unaffected by environmental constraints. For example, in autonomous driving, foggy weather severely restricts camera field of view; excessively low visibility can cause autonomous / assisted driving systems to malfunction, posing safety risks as well.

[0003] The additive noise and nonlinear data-related noise introduced by haze make dehazing a challenging task. As a fundamental low-level computer vision task, image dehazing has attracted increasing attention from scholars and AI companies in the field. Early dehazing methods often involved acquiring multiple images of the same scene, calculating image depth information, and then reconstructing a haze-free image. In comparison, single-image dehazing has more applicable scenarios, leading to the development of a series of algorithms specifically for single-image dehazing. The atmospheric scattering model is widely used to explain the image acquisition process in hazy conditions: I(x) = J(x)t(x) + A(1-t(x)), where I(x) is the hazy image, J(x) is the clear, haze-free image, A represents the atmospheric light value, and t(x) is the transmittance. Most dehazing methods reconstruct a haze-free image by estimating the transmittance t(x) and the atmospheric light value A. For example, the Dark Channel Prior (DCP) algorithm proposed by He Kaiming et al. in 2009 estimates transmittance t(x) by assuming a low intensity value in a certain channel of an outdoor fog-free image. However, when objects in the scene are similar to atmospheric light, such estimation is often inaccurate. With the development of deep learning, a series of image dehazing algorithms based on convolutional neural networks (CNNs) have been proposed. Compared with traditional methods, CNN-based methods obtain fog-free images by indirectly predicting transmittance t(x) or directly regressing it, avoiding the problem of inaccurate physical parameter estimation in traditional methods. Feature maps at different levels carry information at different levels, and fusing feature maps at different levels can enhance feature extraction capabilities. Dilated convolution technology has the advantage of not sacrificing spatial resolution and is widely used for context information aggregation, but it has been shown that using simple dilated convolution in dehazing networks can lead to mesh artifacts. For scenes with uneven fog, estimating fog-free images by treating channel features and pixel-level features equally is often inaccurate. Summary of the Invention

[0004] The purpose of this invention is to provide an image dehazing method based on contextual information aggregation and fused feature attention. Based on deep learning, it uses smooth dilated convolution to replace traditional dilated convolution to reduce the generated grid artifacts. By fusing feature attention mechanism to weight channel attention and pixel-level attention, it solves the problem of inaccurate dehazing image estimation caused by uneven scene haze and uneven channel haze. It adopts an autoencoder structure and uses a contextual information aggregation network to fuse features at different levels.

[0005] The technical solution to achieve the purpose of this invention is: an image dehazing method based on contextual information aggregation and feature attention fusion, the method comprising the following steps:

[0006] Step 1: For foggy images, extract shallow features of the image using an encoder;

[0007] Step 2: Based on shallow features, aggregate contextual information through the information fusion subnet;

[0008] Step 3: Restore the image using the decoder to obtain the dehazed image.

[0009] Furthermore, in step 1, the encoder includes three units connected in sequence. Each unit includes a convolutional layer, an instance normalization layer, and a ReLU activation layer. The output channels of the three unit convolutional layers are all 64, the kernel size is 3, the stride of the first two convolutional layers is 1, and the scale of the output feature map is the same as that of the input image. The stride of the third convolutional layer is 2, which reduces the scale of the feature map to 1 / 2 of the original image.

[0010] Furthermore, step 2, which involves aggregating contextual information based on shallow features through an information fusion subnet, specifically includes:

[0011] Step 2-1: Construct group structures and connect N group structures in sequence to form a multi-level group structure; the last group structure includes a smooth dilated convolution module and a residual feature attention module, and the other group structures include smooth dilated convolution, residual feature attention blocks, instance normalization layer and ReLU activation layer in addition to the smooth dilated convolution and residual feature attention blocks.

[0012] Step 2-2: The image after shallow feature extraction in Step 1 is input into the multi-level group structure. Then, feature maps from M different levels of the group structure are extracted and input into the fusion sub-network for context information aggregation. <N。

[0013] Furthermore, the smooth dilated convolution module in step 2-1 specifically involves adding a separate shared convolutional layer with a kernel size of (2r-1) before the dilated convolution, where r is the dilation rate and r>1.

[0014] Furthermore, the residual feature attention module in step 2-1 includes an upper branch and a lower branch, and the output of the lower branch is element-wise added to the output of the upper branch;

[0015] The upper branch includes a local residual learning unit, a channel attention unit, and a pixel-level attention unit connected in series.

[0016] (1) The residual learning unit consists of two convolutional layers:

[0017] F c =Conv(δ(Conv(F) in ))+F in )

[0018] In the formula, both convolutional layers are convolutions with 64 output channels, F in The input for residual learning is δ, where δ is the ReLU activation function and F is the input. c The output of residual learning is the input of channel attention;

[0019] (4) The channel attention unit is specifically:

[0020] For the input feature map F c Global average pooling is used to incorporate global channel information into the channel descriptor, through the global average pooling function H. p The shape of the feature map is adjusted from C×H×W to C×1×1, resulting in feature map H. p (F c ):

[0021]

[0022] In the formula, H and W represent the width and height of the feature map, respectively, C is the number of channels, and X... c (i,j) represents the pixel at (i,j) in space for each channel;

[0023] The feature map is then passed through two convolutional layers, Conv, sigmoid(σ), and ReLU(δ), to obtain CA. c :

[0024] CA c =σ(Conv(δ(Conv(H) p (F c ))))

[0025] For CA c With input feature map F c Performing element-wise product yields weighted channel features, i.e., channel attention.

[0026]

[0027] (5) The pixel-level attention unit is specifically:

[0028] Channel attention Performing a convolutional activation operation adjusts the shape of the feature map from C×H×W to 1×H×W, resulting in PA:

[0029]

[0030] PA and channel attention Perform element-wise product operation:

[0031]

[0032] The lower branch includes a bottleneck structure consisting of three convolutional layers arranged sequentially: 1×1, 3×3, and 1×1. The 3×3 convolutional layer has 256 channels, and the final 1×1 convolutional layer matches the number of channels to the number of channels for pixel-level attention.

[0033] Furthermore, in step 2-1, N is set to 9, and the dilation rate of the smooth dilated convolution in the 9 consecutive group structures is set to (2,2,2,2,4,4,4,4,1).

[0034] Furthermore, in step 2-2, the feature maps output from the four different levels of the group structure are extracted as input to the fusion subnet, and then linearly combined using importance weights to obtain the combined feature map F. o As shown below:

[0035] (M1, M2, M3, M4)=Conv(cat(F1, F2, F3, F4))

[0036] F o =M1*F1+M2*F2+M3*F3+M4*F4

[0037] In the formula, F1, F2, F3, and F4 represent four feature maps at different levels. cat indicates that the four feature maps are concatenated along the channel dimension to obtain a tensor with 64*4 channels. Conv is a convolution operation used to adjust the number of output channels to 4. The four channels represent the importance of the four different levels of features, corresponding to four weights (M1, M2, M3, M4).

[0038] Furthermore, the four different levels extracted in step 2-2 are layer 0, layer 3, layer 6, and layer 9.

[0039] Furthermore, in step 3, the decoder uses transposed convolution operation with a kernel size of 4 and a stride of 2 to restore the scale, and then performs another convolution with 64 output channels and a convolution with 3 output channels.

[0040] Compared with the prior art, the significant advantages of this invention are:

[0041] 1) This invention proposes a group structure consisting of smooth dilated convolution and residual feature attention blocks. Smooth dilated convolution significantly reduces grid artifacts while maintaining spatial resolution. It integrates other feature attention mechanisms to assign different weights to different image channels and regions. The residual structure can skip some unimportant low-frequency information such as fog. This invention reduces the number of parameters in the deep learning network while achieving good dehazing results.

[0042] 2) This invention adopts an autoencoder architecture, uses an information fusion subnet to aggregate contextual information, calculates the importance weights of the extracted feature layers, and weights the importance weights of the feature maps with the corresponding feature maps to obtain the feature maps after information aggregation. This can significantly improve the feature extraction capability and greatly improve the peak signal-to-noise ratio and structural similarity index of the generated images.

[0043] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the image dehazing method model based on contextual information aggregation and feature attention fusion of the present invention.

[0045] Figure 2 This is a schematic diagram of the shallow feature extraction structure.

[0046] Figure 3 This is a schematic diagram of the residual feature attention module.

[0047] Figure 4 This is a flowchart for channel attention and pixel-level attention.

[0048] Figure 5 A schematic diagram of the group structure composed of smoothly dilated convolutions and fused feature attention blocks.

[0049] Figure 6 A schematic diagram illustrating contextual information aggregation for the information fusion subnet.

[0050] Figure 7 The figures show a comparison of the effects of the present invention and classic image dehazing methods. Figure (a) is the real image, Figure (b) is the fogged image corresponding to Figure (a), and Figures (c) to (g) are the dehazing results of Figure (a) obtained by the DCP method, GCANet method, FFANet method, DehazeFormer method, and the method of the present invention, respectively. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0053] In one embodiment, combined Figure 1 This invention provides an image dehazing method based on contextual information aggregation and feature attention fusion, the method comprising the following steps:

[0054] Step 1: For foggy images, extract shallow features using an encoder. Combined with... Figure 2 The encoder consists of three units connected in sequence. Each unit includes a convolutional layer, an instance normalization layer, and a ReLU activation layer. The output channels of the three convolutional layers are all 64, and the kernel size is 3. The stride of the first two convolutional layers is 1, and the scale of the output feature map is the same as that of the input image. The stride of the third convolutional layer is 2, which reduces the scale of the feature map to 1 / 2 of the original image. The purpose of downsampling is to reduce the amount of computation.

[0055] Step 2: Based on shallow features, aggregate contextual information through an information fusion subnet; specifically including:

[0056] Step 2-1: Construct the group structure and connect the 9 group structures sequentially to form a multi-level group structure, such as... Figure 5 and Figure 6 As shown; the last group structure includes a smooth dilated convolution module and a residual feature attention module. In addition to smooth dilated convolution and residual feature attention, the other group structures also include an instance normalization layer and a ReLU activation layer. The dilation rate of the smooth dilated convolution in the nine consecutive group structures is set to (2,2,2,2,4,4,4,4,1).

[0057] Here, dilated convolution can increase the receptive field without sacrificing spatial resolution by adjusting the dilation rate r (>1). However, severe mesh artifacts occur as the network depth increases. By defining separation and sharing as additional convolution operations, the dependency of input units is increased, which significantly reduces mesh artifacts. The smooth dilated convolution module in step 2-1 is specifically: a separation and sharing convolution layer with a kernel size of (2r-1) is added before the dilated convolution, where r is the dilation rate, and r>1.

[0058] Here, for the dehazing task, different channels have different weighting information. For images with uneven haze, targeted weighting processing is needed for different channels and different image regions. This invention, based on ResNet, employs a fusion feature attention structure. By treating different channel features and pixel regions unequally, it generates different weights for each channel feature and pixel feature, such as... Figure 3 As shown. In step 2-1, the residual feature attention module includes an upper branch and a lower branch. The output of the lower branch is element-wise added to the output of the upper branch.

[0059] The upper branch includes a local residual learning unit, a channel attention unit, and a pixel-level attention unit connected in series. Local residual learning allows less important information such as low-frequency areas like fog to be bypassed, and more attention is paid to effective information.

[0060] (1) The residual learning unit consists of two convolutional layers:

[0061] F c =Conv(δ(Conv(F) in ))+F in )

[0062] In the formula, both convolutional layers are convolutions with 64 output channels, F in The input for residual learning is δ, where δ is the ReLU activation function and F is the input. c The output of residual learning is the input of channel attention;

[0063] (2) Channel attention unit, which assigns different weighted information to different channels, with the structure as follows: Figure 4 Specifically:

[0064] For the input feature map F c Global average pooling is used to incorporate global channel information into the channel descriptor, through the global average pooling function H. p The shape of the feature map is adjusted from C×H×W to C×1×1, resulting in feature map H. p (F c ):

[0065]

[0066] In the formula, H and W represent the width and height of the feature map, respectively, C is the number of channels (taken as 64), and X... c (i,j) represents the pixel at (i,j) in space for each channel;

[0067] The feature map is then passed through two convolutional layers, Conv, sigmoid(σ), and ReLU(δ), to obtain CA. c :

[0068] CA c =σ(Conv(δ(Conv(H) p (F c ))))

[0069] For CA c With input feature map F c Performing element-wise product yields weighted channel features, i.e., channel attention.

[0070]

[0071] (3) Pixel-level attention units: Since the degree of fog may vary in different regions of a foggy image, pixel-level attention is used to focus on key information such as foggy pixels and high-frequency information of the image. Combined with... Figure 4 Specifically:

[0072] Channel attention Performing a convolutional activation operation adjusts the shape of the feature map from C×H×W to 1×H×W, resulting in PA:

[0073]

[0074] Here, the two convolutional layers adjust the number of channels to 8 and 1 respectively;

[0075] PA and channel attention Performing element-wise product operations yields weighted feature maps for different spatial regions:

[0076]

[0077] The lower branch includes a bottleneck structure that enhances the network’s feature extraction capabilities. This bottleneck structure consists of three convolutional layers arranged in sequence: 1×1, 3×3, and 1×1. The number of channels in the 3×3 convolutional layer is set to 256, and the last 1×1 convolutional layer matches the number of channels with the number of channels for pixel-level attention.

[0078] Step 2-2: The image after shallow feature extraction in Step 1 is input into the multi-level group structure. Then, the feature maps output from the four different levels of the group structure are extracted and input into the fusion sub-network. The combined feature map F is obtained by linear combination through importance weights. o As shown below:

[0079] (M1, M2, M3, M4)=Conv(cat(F1, F2, F3, F4))

[0080] F o =M1*F1+M2*F2+M3*F3+M4*F4

[0081] In the formula, F1, F2, F3, and F4 represent four feature maps at different levels, cat indicates concatenation of the four feature maps along the channel dimension to obtain a tensor with 64*4 channels, Conv is a convolution operation used to adjust the number of output channels to 4, and the four channels represent the importance of the features at four different levels, corresponding to four weights (M1, M2, M3, M4). In this invention, feature maps from layers 0, 3, 6, and 9 of the continuous group structure are extracted as input to the fusion subnet, such as... Figure 6 As shown.

[0082] Step 3: Reconstruct the image using the decoder to obtain the dehazed image. Because a downsampling operation was performed during shallow feature extraction to reduce computational complexity, resulting in a 1 / 2 size reduction, scale restoration is required in the decoder. This invention employs transposed convolution in the decoder, with a kernel size of 4 and a stride of 2, to restore scale. This is followed by another convolution with 64 output channels and then another convolution with 3 output channels (in the original image space).

[0083] In one embodiment, an image dehazing system based on contextual information aggregation and feature attention fusion is provided, the system comprising:

[0084] The first module is used to extract shallow features of foggy images through an encoder.

[0085] The second module is used to aggregate contextual information through the information fusion subnet based on shallow features;

[0086] The third module is used to restore the image through the decoder to obtain the dehazed image.

[0087] Specific limitations regarding the image dehazing system based on contextual information aggregation and fused feature attention can be found in the limitations of the image dehazing method based on contextual information aggregation and fused feature attention mentioned above, and will not be repeated here. Each module in the aforementioned image dehazing system based on contextual information aggregation and fused feature attention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0088] Comparing the effects of this invention with classic image dehazing methods, such as... Figure 7As shown, compared with other dehazing methods, the DCP method produces severe color distortion due to the prior assumption of channel-based dehazing; GCANet has a good dehazing effect, but it has a serious color shift and poor preservation of some local details and textures; the FFANet method has a good dehazing effect, but it tends to increase the contrast, and the resulting image is more vivid than the hazy image; the method proposed in this invention has a strong dehazing effect, and the subjective effect is close to the real scene.

[0089] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. An image dehazing method based on contextual information aggregation and feature attention fusion, characterized in that, The method includes the following steps: Step 1: For foggy images, extract shallow features of the image using an encoder; Step 2: Based on shallow features, aggregate contextual information through the information fusion subnet; Step 3: Restore the image using the decoder to obtain the dehazed image; Step 2, which involves aggregating contextual information based on shallow features through an information fusion subnet, specifically includes: Step 2-1: Construct group structures and connect N group structures in sequence to form a multi-level group structure; the last group structure includes a smooth dilated convolution module and a residual feature attention module, and the other group structures include smooth dilated convolution, residual feature attention blocks, instance normalization layer and ReLU activation layer in addition to the smooth dilated convolution and residual feature attention blocks. Step 2-2: The image after shallow feature extraction in Step 1 is input into the multi-level group structure. Then, feature maps from M different levels of the group structure are extracted and input into the fusion sub-network for context information aggregation. <N; The smooth dilated convolution module in step 2-1 is specifically as follows: before the dilated convolution, a separate shared convolutional layer with a kernel size of (2r-1) is added, where r is the dilation rate and r>1; In step 2-1, the residual feature attention module includes an upper branch and a lower branch. The output of the lower branch is added to the output of the upper branch element by element. The upper branch includes a local residual learning unit, a channel attention unit, and a pixel-level attention unit connected in series. (1) The residual learning unit consists of two convolutional layers: F c =Conv(δ(Conv(F in ))+F in ) In the formula, both convolutional layers are convolutions with 64 output channels, F in The input for residual learning is δ, where δ is the ReLU activation function and F is the input. c The output of residual learning is the input of channel attention; (2) The channel attention unit is specifically: For the input feature map F c Global average pooling is used to incorporate global channel information into the channel descriptor, through the global average pooling function H. p The shape of the feature map is adjusted from C×H×W to C×1×1, resulting in feature map H. p (F c ): In the formula, H and W represent the width and height of the feature map, respectively, C is the number of channels, and X... c (i,j) represents the pixel at (i,j) in space for each channel; The feature map is then passed through two convolutional layers, Conv, sigmoid(σ), and ReLU(δ), to obtain CA. c : CA c =σ(Conv(δ(Conv(H p (F c )))) For CA c With input feature map F c Performing element-wise product yields weighted channel features, i.e., channel attention. (3) The pixel-level attention unit is specifically: Channel attention Performing a convolutional activation operation adjusts the shape of the feature map from C×H×W to 1×H×W, resulting in PA: PA and channel attention Perform element-wise product operation: The lower branch includes a bottleneck structure consisting of three convolutional layers arranged sequentially: 1×1, 3×3, and 1×1. The 3×3 convolutional layer has 256 channels, and the final 1×1 convolutional layer matches the number of channels to the number of channels for pixel-level attention.

2. The image dehazing method based on contextual information aggregation and feature attention fusion according to claim 1, characterized in that, In step 1, the encoder consists of three units connected in sequence. Each unit includes a convolutional layer, an instance normalization layer, and a ReLU activation layer. The output channels of the three convolutional layers are all 64, the kernel size is 3, the stride of the first two convolutional layers is 1, and the scale of the output feature map is the same as that of the input image. The stride of the third convolutional layer is 2, which reduces the scale of the feature map to 1 / 2 of the original image.

3. The image dehazing method based on contextual information aggregation and feature attention fusion according to claim 1, characterized in that, In step 2-1, N is set to 9, and the dilation rate of the smooth dilated convolution in the 9 consecutive group structures is set to (2,2,2,2,4,4,4,4,1).

4. The image dehazing method based on contextual information aggregation and feature attention fusion according to claim 3, characterized in that, In step 2-2, the feature maps output from the four different levels of the group structure are extracted as input to the fusion subnet, and then linearly combined using importance weights to obtain the combined feature map F. o As shown below: (M1, M2, M3, M4)=Conv(cat(F1, F2, F3, F4)) F o =M1*F1+M2*F2+M3*F3+M4*F4 In the formula, F1, F2, F3, and F4 represent four feature maps at different levels. cat indicates that the four feature maps are concatenated along the channel dimension to obtain a tensor with 64*4 channels. Conv is a convolution operation used to adjust the number of output channels to 4. The four channels represent the importance of the four different levels of features, corresponding to four weights (M1, M2, M3, M4).

5. The image dehazing method based on contextual information aggregation and feature attention fusion according to claim 4, characterized in that, The four different levels extracted in step 2-2 are level 0, level 3, level 6, and level 9.

6. The image dehazing method based on contextual information aggregation and feature attention fusion according to claim 2, characterized in that, In step 3, the decoder uses transposed convolution with a kernel size of 4 and a stride of 2 to restore the scale. Then, it performs another convolution with 64 output channels and another convolution with 3 output channels.

Images