Attention video defogging method based on flexible space-time region division

Abstract

The application provides an attention video defogging method based on flexible space-time region division, which extracts space-time correlation features in the foggy video frame in a space-time attention manner to enhance the defogging effect of the video defogging algorithm when facing non-uniform fog and large-scale motion scenes. The method uses a sliding window correlation attention module to extract fog density information that interacts in space to solve the problem of fog density changes between different regions. In addition, the attention video defogging method based on flexible space-time region division uses a time complementary information enhancement module to capture complementary information between reference frames and adjacent frames to solve the problem of time changes between video frames and reduce interframe artifacts.

Description

Technical Field

[0001] This invention provides a video defogging method for handling non-uniform fog and large-scale motion scenes. Background Technology

[0002] Videos captured in hazy weather often exhibit severe visibility reduction and color distortion, leading to a significant performance degradation in most downstream visual tasks such as autonomous driving and object detection. Therefore, video dehazing algorithms have been a focus of attention in order to improve the performance of downstream visual tasks in hazy weather.

[0003] Video dehazing algorithms originated from single-frame dehazing algorithms. However, compared to single-frame dehazing, video dehazing not only needs to remove haze from each frame but also needs to maintain the temporal stability of the dehazing results. Early video dehazing methods mainly used convolutional neural networks (CNNs) to predict potential haze-free video frames. However, this short-range modeling method based on CNNs struggles to capture long-range spatial details, resulting in poor performance when dealing with non-uniform haze scenes. To overcome this limitation, recent research has adopted algorithms based on the Swin Transformer, extending the receptive field to a larger attention window to explore and utilize long-range spatial dependencies, thereby enhancing the model's global understanding ability. However, the inflexibility of the Swin Transformer in window partitioning and translation strategies limits long-range interactions between feature points inside and outside the window, leading to the loss of some semantic information. Therefore, designing a method that can address the uneven distribution of haze in video frames remains a major challenge in current video dehazing research.

[0004] To address temporal variations between video frames and produce temporally consistent results, video dehazing algorithms seek to propagate information across time series. Early deep learning-based video dehazing algorithms primarily aggregated temporal information through simple feature fusion methods (such as feature stacking). Feature stacking methods employ a simplistic fusion strategy when processing inter-frame features, failing to adequately consider the correlations between these features. Consequently, it cannot establish long-range dependencies across frames. To establish dependencies based on inter-frame feature correlations, recent video dehazing algorithms have introduced temporal alignment methods to aggregate complementary information across frames. Temporal alignment methods first estimate the offsets of feature points in the reference frame within adjacent frames. Based on these offsets, it selects corresponding feature points from adjacent frames and then performs feature fusion. However, in dynamic scenes with fast-moving objects, such as driving scenes, temporal alignment methods cannot accurately estimate feature point offsets, leading to detail loss due to mismatched feature fusion. Furthermore, estimating the offset of each feature point is computationally complex. For example, using a classic spatial pyramid network, estimating the offset after k iterations has a computational complexity of at least O(n*k), where n is the number of feature points. Therefore, although existing video dehazing algorithms have introduced time alignment methods to aggregate cross-frame information, they still face significant challenges in handling fast-moving objects in dynamic scenes and in terms of computational complexity. Summary of the Invention

[0005] To address the issues of incomplete haze removal in non-uniform haze scenes and inter-frame artifacts in dehazing results during large-scale motion scenes, this invention proposes an attention-based video dehazing method based on flexible spatiotemporal region partitioning. This proposed method significantly reduces non-uniform haze in video frames and minimizes inter-frame flicker artifacts in large-scale motion scenes. Figure 1 As shown, the technical solution of the present invention is based on the construction of a spatiotemporal attention network, and the overall architecture is an encoding and decoding network.

[0006] The training process of the network includes the following steps:

[0007] Step 1: Acquiring pairs of clear video frames and hazy video frames. First, use the camera to acquire consecutive clear video frames J(x). [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] .

[0008] Step two: Input consecutive haze video frames I(x) into a two-dimensional patch embedding layer, transforming the two-dimensional video frames into a one-dimensional patch block feature sequence F. [r-n,r+n] .

[0009] Step 3, convert the feature sequence F [r-n,r+n] The input is fed into the sliding window association attention module to calculate the spatial association information of the feature point relative to the surrounding area. The calculation steps are as follows: For the feature point at (i,j) in the r-th video frame... Constructing in the r-th video frame Centered sliding window Calculate feature points Compared to The calculation steps for the attention value between feature points are as follows: in Depend on Obtained through linear transformation, Depend on After linear transformation, DWConv represents depthwise separable convolution with a kernel size of 3×3.

[0010] Step four: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F. r Compared to the features F of adjacent frames [r-n,r+n] The complementary information between the reference frames is extracted and further incorporated into the reference frame features F. r The specific calculation steps are as follows.

[0011] a. Selecting reference frame feature points In adjacent frame feature sequences F [r-n,r+n] Construct a time neighborhood window Where a = [i - (k / 2), ..., i + (k / 2)], b = [j - (k / 2), ..., j + (k / 2)], and k represents the size of the time neighborhood window.

[0012] b. Calculate the feature points of the reference frame. Compared to the temporal correlation matrix of feature points within the temporal neighborhood window, the process is as follows: Assume the feature points of the reference frame... After linear mapping to Mapping feature points within the temporal neighborhood to a key. and Time correlation matrix The calculation is as follows:

[0013] c. Then, by using the time attention matrix With neighboring Multiplication yields complementary information between features of adjacent frames and those of the reference frame. The calculation method is as follows: Furthermore, the obtained complementary information is fused with the features of the reference frame.

[0014] Step 5: Convert the spatiotemporally encoded video feature sequence F [r-n,r+n] The input sequence is fed into a downsampling layer. The input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is... The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2.

[0015] Step six: After two stages of spatiotemporal coding and downsampling, the obtained spatiotemporal features are input into a temporal memory enhancement module, which uses temporal complementary information and global memory information to enhance the features of the reference frame.

[0016] Step 7: Perform upsampling on the reference frame features. First, use a 2×2 convolution to perform a convolution operation on the reference frame features, so that its number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions, so that the number of channels becomes C.

[0017] Step 8: Using a multi-frame fusion residual connection, the shallow adjacent frame features are fused into the deep reference frame features. First, the shallow adjacent frame features are input into a depthwise separable convolution, and then fused with the deep reference frame features. The calculation method is as follows: in Representing deep reference frame features, This represents the features of adjacent frames in the shallow layer.

[0018] Step nine involves inputting the fused features into a decoder module constructed based on MLP and depthwise separable convolutions. The basic computation steps can be described as follows:

[0019] Step 10: The model's predicted output is the residual R(x) = J(x) - I(x) between the clear video frame and the hazy video frame. Using the input I(x), the model's predicted output J(x) is obtained. pre Calculate J(x) pre The L1 loss between J(x) and J(x) is used to train the model.

[0020] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by a network built on a Transformer architecture. The spatiotemporal encoder of the network comprises three stages, each containing eight sliding window-associated attention modules. In the first stage encoder, only the two deepest sliding window-associated attention modules use spatial self-attention blocks to extract spatial features, while the six shallower modules are replaced with depthwise separable convolutional modules. In the second and third stages, the number of modules using spatial self-attention blocks increases to four and six respectively, to address the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

[0021] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by a sliding window associative attention module comprising two stages. The first stage is a sliding window spatial attention module. First, the video sequence features are input into a layernorm layer and mapped to Q, K, and V values. The sliding window is then divided to calculate the self-attention values ​​of Q, K, and V. Simultaneously, a parallel depthwise separable convolution is used to extract the frequency domain information of V. This is followed by a feedforward layer with an MLP ratio of 2. In the second stage, a sliding window spatial attention module based on an expansion factor is used to expand the originally densely connected small windows into sparsely connected large windows, improving the model's ability to process high-resolution videos without introducing additional computational overhead.

[0022] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by its use of a temporal memory enhancement module, which comprises two parallel stages. Stage one consists of a temporal complementary information enhancement module, whose output for the reference frame is... Phase two consists of a memory enhancement module. This module contains a global memory block that stores additional information from the entire dehazing training set. Using the reference frame features as the query, a query operation is performed within the global memory block. Based on the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

[0023] The advantages of this method are as follows: The proposed attention-based video dehazing method based on flexible spatiotemporal region partitioning uses a sliding window approach to associate attention, allowing each feature point in space to focus on its surrounding spatial region, thus addressing the uneven distribution of haze in space. Furthermore, it extracts and fuses information complementary to the reference frame from adjacent frames through temporal complementary information enhancement, reducing inter-frame flicker artifacts. Therefore, the proposed attention-based video dehazing method based on flexible spatiotemporal region partitioning can better remove haze in unevenly distributed haze and large-scale motion scenes, generating continuous and stable clear video frames. Attached Figure Description

[0024] Appendix Figure 1 This is a schematic diagram of the network model for the attention-based video dehazing method with flexible spatiotemporal region partitioning of the present invention.

[0025] Appendix Figure 2 This is a schematic diagram of the temporal complementary information enhancement module of the attention-based video dehazing method with flexible spatiotemporal region division according to the present invention.

[0026] Appendix Figure 3 This is a schematic diagram of the decoder module of the attention video dehazing method with flexible spatiotemporal region division according to the present invention.

[0027] Appendix Figure 4 The dehazing effect of the proposed flexible spatiotemporal region division video dehazing method is demonstrated. Detailed Implementation

[0028] The following details a typical embodiment of the attention-based video dehazing method based on flexible spatiotemporal region division according to the present invention, providing a further specific description of the method. It is important to note that the following embodiments are only for further illustrative purposes and should not be construed as limiting the scope of protection of the method. Any non-essential improvements and adjustments made to the method by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0029] This invention proposes an attention-based video dehazing method based on flexible spatiotemporal region partitioning, such as... Figure 1 As shown, the basic steps of this method are as follows:

[0030] Step 1: Acquiring pairs of clear video frames and hazy video frames. First, use the camera to acquire consecutive clear video frames J(x). [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] .

[0031] Step two: Input consecutive haze video frames I(x) into a two-dimensional patch embedding layer, transforming the two-dimensional video frames into a one-dimensional patch block feature sequence F. [r-n,r+n] .

[0032] Step 3, convert the feature sequence F [r-n,r+n] The input is fed into the sliding window association attention module to calculate the spatial association information of the feature point relative to the surrounding area. The calculation steps are as follows: For the feature point at (i,j) in the r-th video frame... Constructing in the r-th video frame Centered spatial sliding window Calculate feature points Compared to The calculation steps for the attention value between feature points are as follows: in Depend on Obtained through linear transformation, Depend on After linear transformation, DWConv represents depthwise separable convolution with a kernel size of 3×3.

[0033] Step four: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F. r Compared to the features F of adjacent frames [r-n,r+n] The complementary information between the reference frames is extracted and further incorporated into the reference frame features F. r The specific calculation steps are as follows.

[0034] a. Selecting reference frame feature points In adjacent frame feature sequences F [r-n,r+n] Construct a time neighborhood window Where a = [i - (k / 2), ..., i + (k / 2)], b = [j - (k / 2), ..., j + (k / 2)], and k represents the size of the time neighborhood window.

[0035] b. Calculate the feature points of the reference frame. Compared to the temporal correlation matrix of feature points within the temporal neighborhood window, the process is as follows: Assume the feature points of the reference frame... After linear mapping to Mapping feature points within the temporal neighborhood to a meaning and Time correlation matrix The calculation is as follows:

[0036] c. Then, by using the time attention matrix With neighboring Multiplication yields complementary information between features of adjacent frames and those of the reference frame. The calculation method is as follows: Furthermore, the obtained complementary information is fused with the features of the reference frame.

[0037] Step 5: Convert the spatiotemporally encoded video feature sequence F [r-n,r+n] The input sequence is fed into a downsampling layer. The input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is... The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2.

[0038] Step six: After two stages of spatiotemporal coding and downsampling, the obtained spatiotemporal features are input into a temporal memory enhancement module, which uses temporal complementary information and global memory information to enhance the features of the reference frame.

[0039] Step 7: Perform upsampling on the reference frame features. First, use a 2×2 convolution to perform a convolution operation on the reference frame features, so that its number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions, so that the number of channels becomes C.

[0040] Step 8: Using a multi-frame fusion residual connection, the shallow adjacent frame features are fused into the deep reference frame features. First, the shallow adjacent frame features are input into a depthwise separable convolution, and then fused with the deep reference frame features. The calculation method is as follows: in Representing deep reference frame features, This represents the features of adjacent frames in the shallow layer.

[0041] Step nine involves inputting the fused features into a decoder module constructed based on MLP and depthwise separable convolutions. The basic computation steps can be described as follows:

[0042] Step 10: The model's predicted output is the residual R(x) = J(x) - I(x) between the clear video frame and the hazy video frame. Using the input I(x), the model's predicted output J(x) is obtained. pre Calculate J(x) pre The L1 loss between J(x) and J(x) is used to train the model.

[0043] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by a network built on a Transformer architecture. The spatiotemporal encoder of the network comprises three stages, each containing eight sliding window-associated attention modules. In the first stage encoder, only the two deepest sliding window-associated attention modules use spatial self-attention blocks to extract spatial features, while the six shallower modules are replaced with depthwise separable convolutional modules. In the second and third stages, the number of modules using spatial self-attention blocks increases to four and six respectively, to address the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

[0044] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by a sliding window associative attention module comprising two stages. The first stage is a sliding window spatial attention module. First, the video sequence features are input into a layernorm layer and mapped to Q, K, and V values. The sliding window is then divided to calculate the self-attention values ​​of Q, K, and V. Simultaneously, a parallel depthwise separable convolution is used to extract the frequency domain information of V. This is followed by a feedforward layer with an MLP ratio of 2. In the second stage, a sliding window spatial attention module based on an expansion factor is used to expand the originally densely connected small windows into sparsely connected large windows, improving the model's ability to process high-resolution videos without introducing additional computational overhead.

[0045] The attention-based video dehazing method based on flexible spatiotemporal region partitioning is characterized by its use of a temporal memory enhancement module, which comprises two parallel stages. Stage one consists of a temporal complementary information enhancement module, whose output for the reference frame is... Phase two consists of a memory enhancement module. This module contains a global memory block that stores additional information from the entire dehazing training set. Using the reference frame features as the query, a query operation is performed within the global memory block. Based on the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

[0046] In this invention, the clear video frames used during network training are publicly available from the HazeWorld dataset. First, a depth map estimation algorithm is used to obtain the depth map information of the video frames. Then, based on an atmospheric scattering model, the scattering factor β is set to {0.005, 0.01, 0.02, 0.03}, and the global atmospheric light A ∈ [0.75, 1.0], resulting in a total of 1271 paired video frames. These are further divided into a training set (897 videos) and a test set (374 videos). The model's loss function includes the L1 loss between the predicted video frame and the clear video, and the memory loss L from the memory enhancement module. mem In the first 1 / 5 of the network training phase, the global memory blocks are initialized using memory loss, with λ1 = 0.9 and λ2 = 0.9. mem =0.1. In the last 4 / 5 of the network training stages, we focus on optimizing the L1 loss of the model output, setting λ1 = 0.95 and λ... mem =0.05. The specific training process of the proposed network model is as follows: First, a spatiotemporal encoder consisting of a sliding window association attention module and a temporal complementary information enhancement module is constructed. This encoder comprises three stages, with each stage containing 8 sliding window association attention modules using expansion factors of 4, 2, and 1, and sliding window sizes of 13, 13, and 11, respectively. It also includes a temporal complementary information enhancement module with a window size of 11 and an expansion factor of 2. Next, a memory enhancement module enhances the reference frames using additional information from the training set. Then, a decoder module based on depthwise separable convolutions decodes the reference frames according to the spatiotemporal information extracted by the encoder, further recovering the clear results of the reference frames based on an atmospheric scattering model. The network is optimized using the AdamW optimizer with default parameters of β1 = 0.9 and β2 = 0.999. Furthermore, the initial learning rate and batch size are set to 2 × 10⁻⁶. -4 And 8. A cosine annealing strategy is used to adjust the learning rate from the initial value to 2 × 10⁻⁶. -6 The proposed method was trained on an Nvidia RTX 3090 GPU and an Intel Core™ i7-7820X CPU at 3.60 Hz using the PyTorch framework.

[0047] Table 1 compares the dehazing effects of different video dehazing algorithms on the HazeWorld dataset.

[0048]

[0049] To verify the effectiveness and generalization ability of the proposed method, this invention conducted a thorough comparison on the HazeWorld dataset using a trained network and multiple video dehazing algorithms. Table 1 shows the evaluation results of the proposed method and other video dehazing methods in terms of dehazing performance. According to the results in Table 1, the proposed method significantly outperforms other methods in dehazing performance. Specifically, among all compared methods, MAP-Net has the highest PSNR and SSIM, at 27.12 and 0.9349, respectively. Nevertheless, this invention achieves higher PSNR and SSIM than MAP-Net, at 29.59 and 0.9587, respectively. This is mainly attributed to the sliding window associative attention module, which can perform local attention on the region surrounding the current feature point and achieve global modeling as the window slides, thus simultaneously focusing on both local and global information. Furthermore, the temporal complementary information enhancement module adaptively aggregates inter-frame complementary information using temporal attention, solving the information loss problem in temporal alignment methods at high resolutions. Figure 4 The dehazing effect of this invention on the HazeWorld dataset is intuitively demonstrated. As can be seen from the results, the dehazing algorithm of this invention can better restore clear video frames. Thanks to the enhancement characteristics of the temporal complementary information enhancement module, the proposed network can better restore the details in the video frames and handle the edges of the video frames more finely.

Claims

1. An attention-based video dehazing method based on flexible spatiotemporal region partitioning, characterized in that, Includes the following steps: Step 1: Acquiring pairs of clear video frames and hazy video frames. First, use the camera to acquire consecutive clear video frames J(x). [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] ; Step 2: Input consecutive haze video frames I(x) into a two-dimensional patch embedding layer to transform the two-dimensional video frames into a one-dimensional patch block feature sequence F. [r-n,r+n] ; Step 3, convert the feature sequence F [r-n,r+n] The input is fed into the sliding window association attention module to calculate the spatial association information of the feature point relative to the surrounding area. The calculation steps are as follows: For the feature point at (i,j) in the r-th video frame... Constructing in the r-th video frame Centered spatial sliding window Calculate feature points Compared to windows The calculation steps for the attention value between feature points are as follows: in Depend on Obtained through linear transformation, Depend on The result is obtained through linear transformation, where d represents the feature dimension, B is the learnable bias term, DWConv represents depthwise separable convolution, and the kernel size is 3×3. Step four: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F. r Compared to the features F of adjacent frames [r-n,r+n] The complementary information between the reference frames is extracted and further incorporated into the reference frame features F. r The specific calculation steps are as follows: a. Selecting reference frame feature points In adjacent frame feature sequences F [r-m,r+n] Construct a time neighborhood window Where a = [i-(k / 2),…,i+(k / 2)], b = [j-(k / 2),…,j+(k / 2)], and k represents the size of the time neighborhood window; b. Calculate the feature points of the reference frame. Compared to the temporal attention matrix of feature points within a temporal neighborhood window, the process is as follows: feature points in the reference frame... Linearly mapped to Query The feature points in the time neighborhood are mapped to Keys through a linear transformation. and Value Time Attention Matrix The calculation is as follows: c. Then, by using the time attention matrix Value in the neighborhood Multiplication yields complementary information between features of adjacent frames and those of the reference frame. The calculation method is as follows: Furthermore, the obtained complementary information is fused with the features of the reference frame; Step 5: Convert the spatiotemporally encoded video feature sequence F [r-n,r+n] The input sequence is fed into a downsampling layer. The input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is... The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2; Step six: After two stages of spatiotemporal coding and downsampling, the obtained spatiotemporal features are input into a temporal memory enhancement module, which uses temporal complementary information and global memory information to enhance the features of the reference frame. Step 7: Perform upsampling on the reference frame features. First, use a 2×2 convolution to perform a convolution operation on the reference frame features, so that its number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions, so that the number of channels becomes C. Step 8: Using a multi-frame fusion residual connection, the shallow adjacent frame features are fused into the deep reference frame features. First, the shallow adjacent frame features are input into a depthwise separable convolution, and then fused with the deep reference frame features. The calculation method is as follows: in Representing deep reference frame features, Represents the features of adjacent frames in the shallow layer; Step nine involves inputting the fused features into a decoder module constructed based on MLP and depthwise separable convolutions. The basic computation steps can be described as follows: Step 10: The model's predicted output is the residual R(x) = J(x) - I(x) between the clear video frame and the hazy video frame. Using the input I(x), the model's predicted output J(x) is obtained. pre Calculate J(x) pre The loss between J(x) and the model is used to train the model.

2. The attention-based video dehazing method based on flexible spatiotemporal region division according to claim 1, characterized in that, The dehazing method's network is built on the Transformer architecture. The spatiotemporal encoder part of the network consists of three stages, each containing eight sliding window association attention modules. In the first stage encoder, only the two deepest sliding window association attention modules use spatial self-attention blocks to extract spatial features, while the six shallow modules are replaced by depthwise separable convolutional modules. In the second and third stages, the number of modules using spatial self-attention blocks increases to four and six, respectively, to address the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

3. The attention-based video dehazing method based on flexible spatiotemporal region division according to claim 1, characterized in that, The sliding window associative attention module used consists of two stages. The first stage is a sliding window spatial attention module, which first inputs the video sequence features into a layernorm layer, then maps them to Q, K, and V values, divides the sliding window, calculates the self-attention values ​​of Q, K, and V, and uses a parallel depthwise separable convolution to extract the frequency domain information of V. Next is a feedforward layer with an MLP-ratio of 2. In the second stage of the module, a sliding window spatial attention module based on the expansion factor is used to expand the originally densely connected small window into a sparsely connected large window, which improves the model's ability to process high-resolution videos without introducing additional computational overhead.

4. The attention-based video dehazing method based on flexible spatiotemporal regions according to claim 1, characterized in that, The temporal memory enhancement module used comprises two parallel stages: Stage 1 consists of a temporal complementary information enhancement module, whose output to the reference frame is... Phase two consists of a memory enhancement module containing a global memory block that stores additional information from the entire dehazing training set. Using the reference frame features as the query, a query operation is performed within the global memory block. Based on the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

Images