Image defogging method based on aggregated pixel feature extraction

Abstract

The application belongs to the technical field of image processing, and particularly relates to an image defogging method based on aggregated pixel feature extraction; the method comprises the following steps: obtaining an image to be defogged, inputting the image to a trained image defogging model for processing, and obtaining a clear image; the image defogging model comprises three APBM modules, two RAPBM modules, a convolution layer and a feature fusion layer; the APBM module is composed of eight subunits connected in sequence, and the subunit of the first APBM module is composed of a 3*3 convolution layer and an APB module; the RAPBM module is composed of four subunits connected in sequence, and each subunit is composed of an up-sampling layer, an AFF module and an APB module; the application can improve the generalization ability and the accuracy of the defogging effect while maintaining the high-precision defogging effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image defogging method based on aggregated pixel feature extraction. Background Art

[0002] In adverse weather conditions such as haze and dust, image and video sensing devices such as surveillance cameras, drones, and satellite remote sensing systems often struggle to capture high-quality image information. Images captured in foggy conditions often exhibit blurring, reduced contrast, loss of detail, and degraded color quality. Image quality directly impacts the accuracy, robustness, and reliability of computer vision tasks, making high-quality input images crucial for achieving good computer vision results. However, degraded images collected in foggy conditions can have a significant negative impact on subsequent high-level computer vision tasks. For example, they can affect road condition detection in driver assistance devices or navigation systems, leading to delays in timely route planning and inaccurate distance estimation in transportation tasks, resulting in traffic accidents. They can also complicate target detection and recognition tasks, making it difficult to identify and accurately locate targets in public security investigations. Furthermore, they can complicate engineering surveys, causing loss of target edge and texture details, leading to missed and mis-segmented images in engineering design tasks.

[0003] Therefore, it is of great significance to research effective image dehazing algorithms to improve the quality of foggy images, restore the true scene, and recover the original details and texture features of the images. This also plays an important role in subsequent advanced visual tasks. In practical applications, image dehazing technology plays a key role in traffic safety, monitoring and security, military reconnaissance, and aerospace. By improving image quality, this technology improves the safety performance of transportation systems, strengthens the monitoring system's ability to monitor key areas, and enhances the accuracy of military personnel's analysis of target areas. It also promotes the application of satellite remote sensing in Earth observation. In summary, the results of image dehazing research have not only expanded the research field of computer vision in theory, but also provided powerful solutions in multiple practical application scenarios, promoting technological development in related fields.

[0004] However, existing image dehazing methods are prone to problems such as incomplete restoration of image detail texture features, neglect of contained scene edge information, and poor image generalization ability. Therefore, a new image dehazing method is urgently needed to solve the above problems and improve the quality of the dehazed image. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes an image dehazing method based on aggregated pixel feature extraction, the method comprising: obtaining an image to be dehazed and inputting it into a trained image dehazing model for processing to obtain a clear image; the image dehazing model comprises: three APBM modules, two RAPBM modules, a convolutional layer and a feature fusion layer.

[0006] Preferably, the APBM module is composed of 8 subunits connected in sequence, the subunit of the first APBM module is composed of a 3×3 convolutional layer and an APB module; the subunits of the other two APBM modules are composed of a downsampling layer and an APB module; wherein the APB module includes a fine-grained branch, a coarse-grained branch and a fusion path.

[0007] Furthermore, the processing process of the fine-grained branch on the input features is expressed as:

[0008] P=Linear(Padding(LayerNorm(X)))

[0009]

[0010] Among them, P represents the transformation result of the fine-grained branch input feature X after the initial operation, Linear represents the linear layer, Padding represents the padding operation, LayerNorm represents the layer normalization operation, X represents the input feature, Q (i,j) represents the query vector, K p (i, j) represents the fine-grained branch key value vector, ω (i,j)～P(i,j) Represents the fine-grained branch attention weight.

[0011] Furthermore, the processing process of the coarse-grained branch on the input features is expressed as:

[0012] σ(X)=LayerNorm(AvgPool(GELU(Linear(X))))

[0013]

[0014] Among them, σ(X) represents the transformation result of the coarse-grained branch input feature X after the initial operation, LayerNorm represents the layer normalization operation, AvgPool represents the average pooling operation, GELU represents the GELU activation function, Linear represents the linear layer, X represents the input feature, Q (i,j) represents the query vector, K σ (X) represents the coarse-grained branch key value vector, ω (i,j)～σ(X) Represents the coarse-grained branch attention weight.

[0015] Furthermore, the data processing process of the fusion path is expressed as:

[0016]

[0017] A (i,j)～P(i,j) ,A (i,j)～σ(X) =Split(Attention (i,j) )with size[k 2 ,H p W P ]

[0018] APFE(X (i,j) )=SE(A (i,j)～P(i,j) V P(i,j) )+CBAM(A (i,j)～σ(X) V σ(X) )

[0019] Output=APFE(X (i,j) )+F mlp (LayerNorm(X (i,j) ))

[0020] Among them, Attention (i,j) Represents the attention score matrix, Concat represents the concatenation operation, ω (i,j)～P(i,j) represents the fine-grained branch attention weight, ω (i,j)～σ(X) represents the coarse-grained branch attention weight, B (i,j) represents the bias term, d represents the scaling factor of the stable gradient, Softmax represents the Softmax activation function, A (i,j)～P(i,j) Represents the fine-grained part of the attention matrix, A (i,j)～σ(X) Represents the coarse-grained part of the attention matrix, Split represents the segmentation operation, size[k 2 ,H p W P ] represents the segmentation size, k represents the fine-grained window width, H represents the pooling height, and W represents the pooling width; APFE(X (i,j) ) represents the output characteristics of the APFE module, V P(i,j) represents the fine-grained branch value vector, V σ(X) represents the coarse-grained branch value vector, SE represents the channel compression attention operation, CBAM represents the mixed attention operation, and F mlp Represents the multilayer perceptron operation, LayerNorm represents the layer normalization operation, and Output represents the output feature of the APB module.

[0021] Preferably, the RAPBM module is composed of 4 subunits connected in sequence, each subunit is composed of an upsampling layer, an AFF module and an APB module; the AFF module of the first RAPBM module is jump-connected to the output of the second APBM module, and the AFF module of the second RAPBM module is jump-connected to the output of the first APBM module.

[0022] Furthermore, the data processing process of the AFF module is expressed as follows:

[0023]

[0024]

[0025]

[0026] in, represents the feature after convolution, F(x1) represents the point-by-point convolution of the feature map from the jump connection, x2 represents the feature map from the main path, GAP represents the global average pooling operation, and F mlp Represents the multilayer perceptron operation, Softmax represents the Softmax activation function, Split represents the split operation, a1 represents the first weight, a2 represents the second weight, and y represents the output feature of the AFF module.

[0027] Preferably, the feature fusion layer is used to perform feature fusion on the dehazed image and the output of the 3×3 convolutional layer to output a clear image.

[0028] Preferably, the loss function during image defogging model training is expressed as:

[0029]

[0030] Among them, L1 represents the total loss of the model, N represents the total number of pixels, and I i Represents the pixel value of the i-th pixel in the original image, Represents the pixel value of the i-th pixel in the dehazed image.

[0031] The beneficial effects of the present invention are as follows: the present invention designs an image dehazing model that combines coarse and fine-grained feature extraction with an attention mechanism, which performs fine-grained and coarse-grained attention on different scale features of the image through a dual-path Swin-Transformer backbone network, effectively improving the model's ability to extract image information in a hazy environment. At the same time, the embedded channel compression attention module and hybrid attention module further optimize the image detail recovery and channel feature fusion. By removing redundant residual connections and introducing an adaptive feature fusion module, the model proposed in the present invention can improve the generalization ability and accuracy of the dehazing effect while maintaining a high-precision dehazing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the image defogging model structure in the present invention;

[0033] Figure 2 Schematic diagram of the APB module structure in the present invention;

[0034] Figure 3 Schematic diagram of channel compression attention in the present invention;

[0035] Figure 4 This is a schematic diagram of the AFF module structure in the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] The present invention proposes an image defogging method based on aggregated pixel feature extraction, the method comprising: obtaining an image to be defogged, and inputting the image into a trained image defogging model for processing to obtain a clear image; Figure 1 As shown in Figure 1, the image dehazing model includes three APBM modules, two RAPBM modules, a convolutional layer, and a feature fusion layer.

[0038] The APBM module consists of 8 sub-units connected in sequence. The sub-unit of the first APBM module consists of a 3×3 convolutional layer and an APB module; the sub-units of the other two APBM modules consist of a downsampling layer and an APB module; among them, the APB module includes fine-grained branches, coarse-grained branches and a fusion path.

[0039] The RAPBM module consists of four subunits connected in sequence, each subunit consists of an upsampling layer, an AFF module and an APB module; the AFF module of the first RAPBM module is jump-connected to the output of the second APBM module, and the AFF module of the second RAPBM module is jump-connected to the output of the first APBM module.

[0040] Aggregate pixel feature extraction module (APB):

[0041] As the number of layers in the ViT model increases, the efficiency of information mixing decreases, resulting in degradation of the model's performance. Even by stacking many layers, traces of window partitioning will still form unnatural artifacts in the image. In order to address this problem and restore the detailed texture features of the image, this paper proposes an aggregated pixel feature extraction module to optimize the dehazing network model to improve the model's information perception ability and overall performance. Figure 2 As shown, the aggregated pixel feature extraction module calculates the shape of the input and reshapes the query vector-key value vector-value vector (QKV), calculates the size and length of the pooled features, and expands the local window to calculate the local sequence length and scaling. One path captures local information by calculating similarity within the local window, while the other path captures global information through spatial reduction and pooling, generates pooled features, applies spatial reduction and normalization, generates pooled keys and values, and calculates global similarity. The local and global similarity matrices are concatenated, and attention weights are calculated through normalization. The attention weights are aggregated to the local and pooled feature values ​​respectively, and the input features are weighted summed to output the processed features.

[0042] The APB module includes a fine-grained branch, a coarse-grained branch, and a fusion path. For the fine-grained branch, the input image pair X is used, the position of the center pixel of the window is defined as (i, j), the corresponding window size is set to k×k, and the pixel in the window is defined as p(i, j), so ‖p(i, j)‖=k 2 , given an input feature map, use linear layer projection, and use the window to divide the projection group label. For the coarse-grained branch: In order to improve the generalization and expressiveness of this branch, it is activated by the activation function and then average pooled, and then passed through the normalization layer. The pixel set is defined as σ(X), and the pooling size is defined as H p ×W P , so ‖σ(X)‖=H p W P .

[0043] The operation of aggregate pixel feature extraction can be described as follows:

[0044] P=Linear(Padding(LayerNorm(X)))

[0045]

[0046] σ(X)=LayerNorm(AvgPool(GELU(Linear(X))))

[0047]

[0048]

[0049] A (i,j)～P(i,j) ,A (i,j)～σ(X) =Split(Attention (i,j) )with size[k 2 ,H p W P ]

[0050] APFE(X (i,j) )=SE(A (i,j)～P(i,j) V P(i,j) )+CBAM(A (i,j)～σ(X) V σ(X) )

[0051] Output=APFE(X (i,j) )+F mlp (LayerNorm(X (i,j) ))

[0052] Among them, P represents the transformation result of the fine-grained branch input feature X after the initial operation, Linear represents the linear layer, Padding represents the padding operation, LayerNorm represents the layer normalization operation, X represents the input feature, Q (i,j) represents the query vector, K p (i, j) represents the fine-grained branch key value vector, ω (i,j)～P(i,j) represents the attention weight of the fine-grained branch, σ(X) represents the transformation result of the coarse-grained branch input feature X after the initial operation, AvgPool represents the average pooling operation, GELU represents the GELU activation function, and K σ (X) represents the coarse-grained branch key value vector, ω (i,j)～σ(X) Represents the coarse-grained branch attention weight, Attention (i,j) Represents the attention score matrix, Concat represents the concatenation operation, B (i,j) represents the bias term, d represents the scaling factor of the stable gradient, Softmax represents the Softmax activation function, A (i,j)～P(i,j) Represents the fine-grained part of the attention matrix, A (i,j)～σ(X) Represents the coarse-grained part of the attention matrix, Split represents the segmentation operation, size[k 2 ,H p W P ] represents the segmentation size, k represents the fine-grained window width, H represents the pooling height, and W represents the pooling width;

[0053] APFE(X (i,j) ) represents the output characteristics of the APFE module, V P(i,j) represents the fine-grained branch value vector, V σ(X)represents the coarse-grained branch value vector, SE represents the channel compression attention operation, CBAM represents the mixed attention operation, and F mlp Represents the multilayer perceptron operation, and Output represents the output features of the APB module.

[0054] By improving the dual-path module of aggregated pixel feature extraction that adapts to fog characteristics, image details can be effectively processed from both coarse-grained and fine-grained aspects. In addition, a specific attention mechanism is embedded in the output of each branch. Specifically, the fine-grained branch focuses on processing finer and higher-frequency image details, especially texture features. Therefore, this branch embeds a channel compression attention mechanism, which can accurately focus on key detail features. By allocating attention to different channels, it enhances the ability to capture and restore local details, such as Figure 3 As shown in the figure. The degree of impact of haze on different areas of the image may vary, and objects near and far are affected differently. The fine-grained branch can focus on local areas of the image, effectively addressing problems such as blurred edges and texture loss caused by haze. During the dehazing process, the fine-grained branch can help the model recover these fine features, while the channel compression attention mechanism further improves the dehazing effect. Since fine-grained features usually carry the most critical edge, texture, and structural information in the image, they play a vital role in the dehazing effect. In addition, the coarse-grained branch is embedded with a hybrid attention module to help the model focus on important areas more accurately when processing large-scale, low-frequency features. By combining channel and spatial attention, the coarse-grained branch can effectively capture the global structural information and spatial distribution in the image, thereby improving the understanding of the overall features, especially in removing large-scale haze.

[0055] Adaptive feature fusion module (FFA) based on selective convolution kernel:

[0056] Traditional feature fusion simply concatenates or sums multi-dimensional features, but these options can only provide limited expression capabilities for the network. The present invention uses an adaptive feature fusion module based on selective convolution kernels to perform nonlinear processing on fusion features of different scales and dynamically adjust the receptive field, such as Figure 4 This feature fusion method is of great significance for enhancing the receptive field and semantic information at different scales. It is adjusted through learnable weight parameters to adapt to the reconstruction requirements of different levels of detail, retaining the original input information, helping the model to better learn and reconstruct fine texture details. The dynamic introduction of features at different levels also helps alleviate the gradient vanishing problem and promotes faster network convergence.

[0057] Suppose there are two feature maps, x1 and x2, where x1 is the feature map from the skip connection and x2 is the feature map from the main path. First, x1 is projected using point-wise convolution. Then, weights are obtained using global average pooling, multi-layer perceptron convolution, normalization, and segmentation. The adaptive weights are used to recalibrate and aggregate the features. The operation of the FFA module is described as follows:

[0058]

[0059]

[0060]

[0061] in, represents the feature after convolution, F(x1) represents the point-by-point convolution of the feature map from the jump connection, x2 represents the feature map from the main path, GAP represents the global average pooling operation, and F mlp Represents the multilayer perceptron operation, Split represents the segmentation operation, a1 represents the first weight, a2 represents the second weight, and y represents the output feature of the AFF module.

[0062] The feature fusion layer is used to perform feature fusion on the image to be dehazed and the output of the convolution layer. Preferably, the feature fusion layer jump-connects the output of the 3×3 convolution layer with the image to be dehazed to output a clear image.

[0063] During the image dehazing model training process, the present invention uses the L1 loss function to train the model. The L1 loss is also called the mean absolute error and is a commonly used loss function for regression tasks. It compares the difference between the predicted output and the target image pixel by pixel, calculates their absolute error, and then obtains the average value. In the image dehazing task, the L1 loss helps to measure the difference between the dehazed image and the real image pixel by pixel, and then optimizes the output of the model so that the generated dehazed image is closer to the real image. At the same time, the L1 loss is more effective in maintaining image clarity and details, and is suitable for processing image generation problems with blur and subtle differences. Therefore, it is often used in tasks such as image dehazing. The L1 loss function is as follows:

[0064]

[0065] Among them, L1 represents the total loss of the model, N represents the total number of pixels, and I i Represents the pixel value of the i-th pixel in the original image, Represents the pixel value of the i-th pixel in the dehazed image.

[0066] After the image defogging model training is completed, the image to be defogged is taken and input into the trained image defogging model for processing, and a clear image after defogging is output.

[0067] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image defogging method based on aggregated pixel feature extraction, characterized in that: include: Obtain the image to be dehazed and input it into the trained image dehazing model for processing to obtain a clear image; The image dehazing model includes: three APBM modules, two RAPBM modules, a convolutional layer and a feature fusion layer; The APBM module consists of 8 subunits connected in sequence. The subunit of the first APBM module consists of a 3×3 convolutional layer and an APB module. The subunits of the other two APBM modules consist of a downsampling layer and an APB module. The APB module includes a fine-grained branch, a coarse-grained branch, and a fusion path. The processing process of the fine-grained branch on the input features is expressed as: P=Linear(Padding(LayerNorm(X))) Among them, P represents the transformation result of the fine-grained branch input feature X after the initial operation, Linear represents the linear layer, Padding represents the padding operation, LayerNorm represents the layer normalization operation, X represents the input feature, Q (i,j) represents the query vector, K p (i, j) represents the fine-grained branch key value vector, ω (i,j)～P(i,j) Represents the fine-grained branch attention weight; The processing process of the coarse-grained branch on the input features is expressed as: σ(X)=LayerNorm(AvgPool(GELU(Linear(X)))) Among them, σ(X) represents the transformation result of the coarse-grained branch input feature X after the initial operation, AvgPool represents the average pooling operation, GELU represents the GELU activation function, and Q (i,j) represents the query vector, K σ (X) represents the coarse-grained branch key value vector, ω (i,j)～σ(X) Represents the coarse-grained branch attention weight; The data processing process of the fusion path is expressed as: A (i,j)～P(i,j) ,A (i,j)～σ(X) =Split(Attention (i,j) )with size[k 2 ,H p W P ] APFE(X (i,j) )=SE(A (i,j)～P(i,j) V P(i,j) )+CBAM(A (i,j)～σ(X) V σ(X) ) Output=APFE(X (i,j) )+F mlp (LayerNorm(X (i,j) )) Among them, Attention (i,j) represents the attention score matrix, Concat represents the concatenation operation, B (i,j) represents the bias term, d represents the scaling factor of the stable gradient, Softmax represents the Softmax activation function, A (i,j)～P(i,j) Represents the fine-grained part of the attention matrix, A (i,j)～σ(X) Represents the coarse-grained part of the attention matrix, Split represents the segmentation operation, size[k 2 ,H p W P ] represents the segmentation size, k represents the fine-grained window width, H p Represents the pooling height, W P Represents the pooling width; APFE(X (i,j) ) represents the output characteristics of the APFE module, V P(i,j) represents the fine-grained branch value vector, V σ(X) represents the coarse-grained branch value vector, SE represents the channel compression attention operation, CBAM represents the mixed attention operation, and F mlp Represents the multilayer perceptron operation, and Output represents the output features of the APB module; The RAPBM module is composed of four subunits connected in sequence, each subunit consists of an upsampling layer, an AFF module and an APB module; the AFF module of the first RAPBM module is skip-connected to the output of the second APBM module, and the AFF module of the second RAPBM module is skip-connected to the output of the first APBM module; The data processing process of the AFF module is expressed as follows: in, represents the feature after convolution, F(x1) represents the point-by-point convolution of the feature map from the jump connection, x2 represents the feature map from the main path, GAP represents the global average pooling operation, a1 represents the first weight, a2 represents the second weight, and y represents the output feature of the AFF module.

2. The image defogging method based on aggregated pixel feature extraction according to claim 1, characterized in that: The feature fusion layer is used to perform feature fusion on the dehazed image and the output of the 3×3 convolutional layer to output a clear image.

3. The image defogging method based on aggregated pixel feature extraction according to claim 1, characterized in that: The loss function during image dehazing model training is expressed as: Among them, L1 represents the total loss of the model, N represents the total number of pixels, and I i Represents the pixel value of the i-th pixel in the original image, Represents the pixel value of the i-th pixel in the dehazed image.

Images