Attention video defogging method based on flexible space-time region division

Abstract

The invention provides an attention video defogging method based on flexible space-time region division, and the method employs a space-time attention mode to extract space-time correlation features in a haze video frame, so as to enhance the defogging effect of a video defogging algorithm when facing non-uniform haze and large-range motion scenes. According to the method provided by the invention, a sliding window is used for associating an attention module to extract haze density information interacting in space, so that the problem of haze density change between different areas is solved. Besides, the attention video defogging method based on flexible space-time region division uses a time complementary information enhancement module to capture complementary information between a reference frame and an adjacent frame, so that the problem of time change between video frames is solved, and inter-frame artifacts are reduced.

Description

Technical Field

[0001] The present invention is a video defogging method for processing non-uniform haze and large-scale motion scenes. Background Art

[0002] Videos collected in hazy weather often show severe visibility reduction and color distortion, which leads to significant performance degradation of most downstream visual tasks such as autonomous driving and object detection. Video dehazing algorithms have been attracting attention by improving the performance of downstream visual tasks in hazy weather.

[0003] Video dehazing algorithms are derived from single image dehazing algorithms. However, compared with single image dehazing, video dehazing not only needs to remove haze from each frame, but also needs to maintain the temporal stability of the dehazing results. Early video dehazing methods mainly used convolutional neural networks (CNNs) to predict potential haze-free video frames. However, this short-range modeling method based on CNNs has difficulty in capturing long-range spatial details, resulting in poor performance when dealing with non-uniform haze scenes. To overcome this limitation, recent studies have adopted algorithms based on Swin Transformer to expand the receptive field to a larger attention window to explore and exploit long-range spatial dependencies, thereby enhancing the global understanding ability of the model. However, the inflexibility of Swin Transformer in window division and translation strategies limits the long-range interaction between feature points inside and outside the window, resulting in the loss of some semantic information. Therefore, designing a method that can address the uneven distribution of haze in video frames remains a major challenge in current video dehazing research.

[0004] To address the temporal variations between video frames and produce temporally consistent results, video dehazing algorithms seek to propagate information in temporal sequences. Early deep learning-based video dehazing algorithms mainly aggregated temporal information through simple feature fusion methods such as feature stacking. Feature stacking methods adopt a simple fusion strategy when processing inter-frame features without fully considering the correlation between these features. Therefore, it cannot establish long-range dependencies across frames. In order to establish dependencies based on inter-frame feature correlation, recent video dehazing algorithms have introduced temporal alignment methods to aggregate complementary information across frames. Temporal alignment methods first estimate the offsets of feature points of the reference frame in adjacent frames. Based on these offsets, it selects corresponding feature points from adjacent frames and then performs feature fusion. However, in dynamic scenes with fast-moving objects, such as in driving scenes, temporal alignment methods cannot accurately estimate feature point offsets, resulting in loss of details due to mismatched feature fusion. In addition, estimating the offset of each feature point is computationally complex. For example, using a classic spatial pyramid network, estimating the offset after k iterations has a computational complexity of at least O(n*k), where n is the number of feature points. Therefore, although existing video dehazing algorithms introduce temporal alignment methods to aggregate cross-frame information, they still face significant challenges in handling fast-moving objects in dynamic scenes and computational complexity. Summary of the invention

[0005] In order to solve the problem of incomplete haze removal in non-uniform haze scenes and inter-frame artifacts in defogging results in large-scale motion scenes, the present invention proposes an attention video defogging method based on flexible spatiotemporal region division. The proposed video defogging method can greatly remove non-uniform haze in video frames and reduce inter-frame flickering artifacts in large-scale motion scenes. Figure 1 As shown, the technical solution of the present invention is constructed based on a spatiotemporal attention network, and the overall architecture is a codec network.

[0006] The training process of the network includes the following steps:

[0007] Step 1: Paired clear video frames and haze video frames are collected. First, the camera is used to collect continuous clear video frames J(x) [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] .

[0008] Step 2: Input the continuous haze video frames I(x) into a two-dimensional patch embedding layer, and convert the two-dimensional video frames into a one-dimensional patch block feature sequence F [r-n,r+n] .

[0009] Step 3: transform the feature sequence F [r-n,r+n] Input to the sliding window correlation attention module to calculate the spatial correlation information of the feature point compared to the surrounding area. The calculation steps are as follows. For the feature point at (i, j) in the rth video frame In the rth video frame, construct The sliding window at the center Calculate feature points Compared to The attention value between the feature points in the equation is calculated as follows: in Depend on After linear transformation, we get Depend on After linear transformation, DWConv stands for depth-wise separable convolution, and the size of the convolution kernel is 3×3.

[0010] Step 4: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F r Compared with the adjacent frame features F [r-n,r+n] The complementary information between the two frames is extracted and further integrated into the reference frame feature F r The specific calculation steps are as follows.

[0011] a. Select reference frame feature points In the adjacent frame feature sequence F [r-n,r+n] Construct a time neighborhood window in Where a = [i-(k / 2), ..., i+(k / 2)], b = [j-(k / 2), ..., j+(k / 2)], where k represents the size of the temporal neighborhood window.

[0012] b. Calculate the reference frame feature points Compared with the time correlation matrix of the feature points in the time neighborhood window, the process is as follows: let the feature points of the reference frame After linear mapping, Map feature points in the time neighborhood to key and Time Correlation Matrix The calculation is as follows:

[0013] c. Then, by transforming the temporal attention matrix With the neighborhood Multiply to obtain the complementary information between the features of adjacent frames compared to the reference frame The calculation method is: The complementary information is further fused with the reference frame features.

[0014] Step 5: Transform the spatiotemporally encoded video feature sequence F [r-n,r+n] Input to a downsampling layer, the input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2.

[0015] Step 6: After two stages of spatiotemporal encoding and downsampling, the spatiotemporal features are input into a temporal memory enhancement module, and the reference frame features are enhanced using temporal complementary information and global memory information.

[0016] Step 7: Upsample the reference frame features. First, use a 2×2 convolution to convolve the reference frame features so that the number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions and the number of channels becomes C.

[0017] Step 8: Use a multi-frame fusion residual connection to fuse the shallow adjacent frame features into the deep reference frame features. First, input the shallow adjacent frame features into a depth-separable convolution, and then fuse them with the deep reference frame features. The calculation method is: in represents the deep reference frame features, Represents shallow adjacent frame features.

[0018] Step 9: Input the fused features into a decoder module based on MLP and depth-wise separable convolution. The basic calculation steps can be described as:

[0019] Step 10: The model predicts the residual R(x)=J(x)-I(x) between the clear video frame and the haze video frame. Using the input I(x), the model predicts the output J(x). pre , calculate J(x) pre The L1 loss between y(x) and J(x) is used to train the model.

[0020] The attention video defogging method based on flexible spatiotemporal region division is characterized in that the network is built based on the Transformer architecture, and the spatiotemporal encoder part of the network includes three stages, and each stage includes 8 sliding window associated attention modules. In the encoder of the first stage, there are only two sliding window associated attention modules in the deepest layer, and spatial self-attention blocks are used to extract spatial features. Depth-separable convolution modules are used instead in the 6 shallow modules. In the second and third stages, the number of modules using spatial self-attention blocks is increased to 4 and 6 respectively to deal with the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

[0021] The attention video dehazing method based on flexible spatiotemporal region division is characterized in that the sliding window associated attention module used contains two stages. The first stage is a sliding window spatial attention module. First, the video sequence features are input into a layernorm layer, and then mapped to Q, K, and V values. The sliding window is divided to calculate the self-attention values ​​of Q, K, and V, and a parallel depth-separable convolution is used to extract the frequency domain information of V. Next is a feedforward layer with an MLP ratio value of 2. In the second stage of the module, a sliding window spatial attention module based on an expansion factor is used to expand the originally densely connected small window into a sparsely connected large window, which improves the model's processing capability for high-resolution videos without introducing additional computational overhead.

[0022] The attention video defogging method based on flexible spatiotemporal region division is characterized in that the temporal memory enhancement module used includes two parallel stages. Stage one consists of a temporal complementary information enhancement module, and the output of the temporal complementary information enhancement module for the reference frame is The second stage consists of a memory enhancement module. The memory enhancement module contains a global memory block, which stores additional information from the entire defogging training set. The reference frame features are used as queries to perform query operations in the global memory block. According to the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

[0023] The beneficial effects of this method are as follows: the proposed attention video defogging method based on flexible spatiotemporal region division allows each feature point in space to focus on the spatial region around itself through sliding window association attention to cope with the situation where haze is unevenly distributed in space; through the method of temporal complementary information enhancement, information complementary to the reference frame is extracted from adjacent frames and fused to reduce flicker artifacts between frames. Therefore, the proposed attention video defogging method based on flexible spatiotemporal region division can better remove haze in space and generate continuous, stable and clear video frames in scenes with uneven haze distribution and large-scale motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Attached Figure 1 Schematic diagram of the network model of the attention video dehazing method with flexible spatiotemporal region division of the present invention.

[0025] Attached Figure 2 Schematic diagram of the temporal complementary information enhancement module of the attention video defogging method with flexible spatiotemporal region division of the present invention.

[0026] Attached Figure 3 Schematic diagram of the decoder module of the attention video defogging method with flexible spatiotemporal region division of the present invention.

[0027] Attached Figure 4 Demonstration of the dehazing effect of the proposed video dehazing method with flexible spatiotemporal region partitioning. DETAILED DESCRIPTION

[0028] A typical embodiment of the attention video defogging method based on flexible spatiotemporal region division of the present invention is described in detail below, and the method is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the method and cannot be understood as limiting the protection scope of the method. Personnel skilled in the art in the field may make some non-essential improvements and adjustments to the method based on the above content of the method, which still fall within the protection scope of the present invention.

[0029] The present invention proposes an attention video defogging method based on flexible spatiotemporal region division, such as Figure 1 As shown, the basic steps of this method are as follows:

[0030] Step 1: Paired clear video frames and haze video frames are collected. First, the camera is used to collect continuous clear video frames J(x) [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] .

[0031] Step 2: Input the continuous haze video frames I(x) into a two-dimensional Patch Embedding layer, and convert the two-dimensional video frames into a one-dimensional patch block feature sequence F [r-n,r+n] .

[0032] Step 3: transform the feature sequence F [r-n,r+n] Input to the sliding window correlation attention module to calculate the spatial correlation information of the feature point compared to the surrounding area. The calculation steps are as follows. For the feature point at (i, j) in the rth video frame In the rth video frame, construct The spatial sliding window centered Calculate feature points Compared to The attention value between the feature points in the equation is calculated as follows: in Depend on After linear transformation, we get Depend on After linear transformation, DWConv stands for depth-wise separable convolution, and the size of the convolution kernel is 3×3.

[0033] Step 4: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F r Compared with the adjacent frame features F [r-n,r+n] The complementary information between the two frames is extracted and further integrated into the reference frame feature F r The specific calculation steps are as follows.

[0034] a. Select reference frame feature points In the adjacent frame feature sequence F [r-n,r+n] Construct a time neighborhood window in Where a = [i-(k / 2), ..., i+(k / 2)], b = [j-(k / 2), ..., j+(k / 2)], where k represents the size of the temporal neighborhood window.

[0035] b. Calculate the reference frame feature points Compared with the time correlation matrix of the feature points in the time neighborhood window, the process is as follows: let the feature points of the reference frame After linear mapping, Map the feature points in the time neighborhood into and Time Correlation Matrix The calculation is as follows:

[0036] c. Then, by transforming the temporal attention matrix With the neighborhood Multiply to obtain the complementary information between the features of adjacent frames compared to the reference frame The calculation method is: The complementary information is further fused with the reference frame features.

[0037] Step 5: Transform the spatiotemporally encoded video feature sequence F [r-n,r+n] Input to a downsampling layer, the input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2.

[0038] Step 6: After two stages of spatiotemporal encoding and downsampling, the spatiotemporal features are input into a temporal memory enhancement module, and the reference frame features are enhanced using temporal complementary information and global memory information.

[0039] Step 7: Upsample the reference frame features. First, use a 2×2 convolution to convolve the reference frame features so that the number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions and the number of channels becomes C.

[0040] Step 8: Use a multi-frame fusion residual connection to fuse the shallow adjacent frame features into the deep reference frame features. First, input the shallow adjacent frame features into a depth-separable convolution, and then fuse them with the deep reference frame features. The calculation method is: in represents the deep reference frame features, Represents shallow adjacent frame features.

[0041] Step 9: Input the fused features into a decoder module based on MLP and depth-wise separable convolution. The basic calculation steps can be described as:

[0042] Step 10: The model predicts the residual R(x)=J(x)-I(x) between the clear video frame and the haze video frame. Using the input I(x), the model predicts the output J(x). pre , calculate J(x) pre The L1 loss between y(x) and J(x) is used to train the model.

[0043] The attention video defogging method based on flexible spatiotemporal region division is characterized in that the network is built based on the Transformer architecture, and the spatiotemporal encoder part of the network includes three stages, and each stage includes 8 sliding window associated attention modules. In the encoder of the first stage, there are only two sliding window associated attention modules in the deepest layer, and spatial self-attention blocks are used to extract spatial features. Depth-separable convolution modules are used instead in the 6 shallow modules. In the second and third stages, the number of modules using spatial self-attention blocks is increased to 4 and 6 respectively to deal with the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

[0044] The attention video dehazing method based on flexible spatiotemporal region division is characterized in that the sliding window associated attention module used contains two stages. The first stage is a sliding window spatial attention module. First, the video sequence features are input into a layernorm layer, and then mapped to Q, K, and V values. The sliding window is divided to calculate the self-attention values ​​of Q, K, and V, and a parallel depth-separable convolution is used to extract the frequency domain information of V. Next is a feedforward layer with an MLP ratio value of 2. In the second stage of the module, a sliding window spatial attention module based on an expansion factor is used to expand the originally densely connected small window into a sparsely connected large window, which improves the model's processing capability for high-resolution videos without introducing additional computational overhead.

[0045] The attention video defogging method based on flexible spatiotemporal region division is characterized in that the temporal memory enhancement module used includes two parallel stages. Stage one consists of a temporal complementary information enhancement module, and the output of the temporal complementary information enhancement module for the reference frame is The second stage consists of a memory enhancement module. The memory enhancement module contains a global memory block, which stores additional information from the entire defogging training set. The reference frame features are used as queries to perform query operations in the global memory block. According to the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

[0046] In the example of the present invention, the clear video frames used in network training are publicly available from the data set HazeWorld. First, the depth map estimation algorithm is used to obtain the depth map information of the video frame. Then, based on the atmospheric scattering model, the scattering factor β is set to {0.005, 0.01, 0.02, 0.03}, and the global atmospheric light A∈[0.75, 1.0] is set to obtain a total of 1271 paired video frames, which are further divided into a training set (897 videos) and a test set (374 videos). The loss function of the model includes the L1 loss between the predicted video frame and the clear video, the memory loss L of the memory enhancement module, and the L2 loss between the predicted video frame and the clear video. mem In the first 1 / 5 of the network training, the global memory block is initialized using memory loss, with λ1 = 0.9, λ mem =0.1. In the last 4 / 5 stages of network training, we focus on optimizing the L1 loss of the model output, setting λ1 = 0.95, λ mem =0.05. The specific training process of the proposed network model is as follows. First, a spatiotemporal encoder consisting of a sliding window association attention module and a temporal complementary information enhancement module is used. It contains three stages. The number of sliding window association attention modules contained in each stage is 8, and the expansion factors used are 4, 2, and 1 respectively. The sizes of the sliding windows are 13, 13, and 11 respectively, including a temporal complementary information enhancement module with a window size of 11 and an expansion factor of 2. Next is a memory enhancement module that uses additional information from the training set to enhance the reference frame. Then there is a decoder module based on a deep separable convolution structure, which decodes the reference frame according to the spatiotemporal information extracted by the encoder, and further restores the clear result of the reference frame according to the atmospheric scattering model. The network is optimized using the AdamW optimizer with the default parameters: β1=0.9 and β2=0.999. In addition, the initial learning rate and batch size are set to 2×10 -4 and 8. The cosine annealing strategy is used to adjust the learning rate from the initial value to 2×10 -6 The training environment of the proposed method is the Pytorch framework with Nvidia RTX 3090GPU and Intel Core(TM)i7-7820X CPU@3.60Hz.

[0047] Table 1 Comparison of dehazing effects of different video dehazing algorithms on the HazeWorld dataset

[0048]

[0049] In order to verify the effectiveness and generalization ability of the proposed method, the present invention uses a trained network and multiple video defogging algorithms to conduct a full comparison on the HazeWorld dataset. Table 1 shows the defogging effect evaluation results of the method proposed in the present invention and other video defogging methods. According to the results in Table 1, the defogging performance of the proposed method is significantly better than other methods. Specifically, among all the compared methods, MAP-Net has the highest PSNR and SSIM, which are 27.12 and 0.9349 respectively. Despite this, the present invention achieves higher PSNR and SSIM than MAP-Net, which are 29.59 and 0.9587 respectively. This is mainly due to the sliding window association attention module, which can pay local attention to the surrounding area of ​​the current feature point and realize global modeling when the window slides, thereby paying attention to local and global information at the same time. In addition, the temporal complementary information enhancement module uses temporal attention to adaptively aggregate complementary information between frames, solving the information loss problem in the temporal alignment method at large resolution. Figure 4 The dehazing effect of the present invention on the HazeWorld dataset is intuitively demonstrated. According to the graphical results, it can be seen that the dehazing algorithm of the present invention can better restore clear video frames. Thanks to the enhanced characteristics of the temporal complementary information enhancement module, the proposed network can better restore the details in the video frames and process the edges of the video frames more finely.

Claims

1. An attention video defogging method based on flexible spatiotemporal region division, characterized in that: The following steps are involved: Step 1: Paired clear video frames and haze video frames are collected. First, the camera is used to collect continuous clear video frames J(x) [r-n,r+n] Then, based on the atmospheric scattering model I(x)=J(x)t(x)+A(1-t(x)), the corresponding haze video I(x) is generated. [r-n,r+n] ; Step 2: Input the continuous haze video frames I(x) into a two-dimensional Patch Embedding layer to convert the two-dimensional video frames into a one-dimensional patch block feature sequence F [r-n,r+n] ; Step 3: transform the feature sequence F [r-n,r+n] Input into the sliding window association attention module to calculate the spatial association information of the feature point compared to the surrounding area. The calculation steps are as follows: For the feature point at (i, j) in the rth video frame In the rth video frame, construct The spatial sliding window centered Calculate feature points Compared to window The attention value between the feature points in the equation is calculated as follows: in Depend on After linear transformation, we get Depend on After linear transformation, DWConv represents depthwise separable convolution, and the size of the convolution kernel is 3×3; Step 4: The feature sequence is further input into a temporal complementary information enhancement module to extract the reference frame feature F r Compared with the adjacent frame features F [r-n,r+n] The complementary information between the two frames is extracted and further integrated into the reference frame feature F r The specific calculation steps are: a. Select reference frame feature points In the adjacent frame feature sequence F [r-n,r+n] Construct a time neighborhood window in Where a = [i-(k / 2), ..., i+(k / 2)], b = [j-(k / 2), ..., j+(k / 2)], where k represents the size of the temporal neighborhood window; b. Calculate the reference frame feature points Compared with the time correlation matrix of the feature points in the time neighborhood window, the process is as follows: let the feature points of the reference frame After linear mapping, Map feature points in the time neighborhood to key and Time Correlation Matrix The calculation is as follows: c. Then, by transforming the temporal attention matrix With the neighborhood Multiply to obtain the complementary information between the features of adjacent frames compared to the reference frame The calculation method is: And further fuse the obtained complementary information with the reference frame features; Step 5: Transform the spatiotemporally encoded video feature sequence F [r-n,r+n] Input to a downsampling layer, the input dimension of the sequence is H×W×C×T, where T represents the time dimension, and the feature dimension after downsampling is The downsampling layer consists of a convolutional layer with a kernel size of 2 and a stride of 2; Step 6: After two stages of spatiotemporal encoding and downsampling, the obtained spatiotemporal features are input into a temporal memory enhancement module, and the reference frame features are enhanced using temporal complementary information and global memory information; Step 7: Upsample the reference frame features. First, use a 2×2 convolution to convolve the reference frame features so that the number of channels becomes 4C. Then, use the PixelShuffle operation to double the width and height dimensions and the number of channels becomes C. Step 8: Use a multi-frame fusion residual connection to fuse the shallow adjacent frame features into the deep reference frame features. First, input the shallow adjacent frame features into a depth-separable convolution, and then fuse them with the deep reference frame features. The calculation method is: in represents the deep reference frame features, Represents shallow adjacent frame features; Step 9: Input the fused features into a decoder module based on MLP and depth-wise separable convolution. The basic calculation steps can be described as: Step 10: The model predicts the residual R(x)=J(x)-I(x) between the clear video frame and the haze video frame. Using the input I(x), the model predicts the output J(x). pre , calculate J(x) pre The loss between y(x) and J(x) is used to train the model.

2. The attention video defogging method based on flexible spatiotemporal region division according to claim 1 is characterized in that: The network is built based on the Transformer architecture. The spatiotemporal encoder part of the network includes three stages, and each stage contains 8 sliding window associated attention modules. In the encoder of the first stage, only the two deepest sliding window associated attention modules use spatial self-attention blocks to extract spatial features, and the 6 shallow modules are replaced by depthwise separable convolution modules. In the second and third stages, the number of modules using spatial self-attention blocks is increased to 4 and 6 respectively to deal with the problem that shallow attention blocks cannot achieve focused attention in space and increase computational complexity.

3. The attention video defogging method based on flexible spatiotemporal region division according to claim 1 is characterized in that: The sliding window associative attention module used consists of two stages. The first stage is a sliding window spatial attention module. The video sequence features are first input into a layernorm layer, and then mapped to Q, K, and V values. The sliding window is divided to calculate the self-attention values ​​of Q, K, and V, and a parallel depth-wise separable convolution is used to extract the frequency domain information of V. Next is a feedforward layer with an MLP-ratio value of 2. In the second stage of the module, a sliding window spatial attention module based on an expansion factor is used to expand the originally densely connected small window into a sparsely connected large window, which improves the model's processing capabilities for high-resolution videos without introducing additional computational overhead.

4. The method for defogging video based on flexible spatiotemporal regions according to claim 1, characterized in that: The temporal memory enhancement module used consists of two parallel stages: the first stage consists of a temporal complementary information enhancement module, the output of which for the reference frame is The second stage consists of a memory enhancement module, which contains a global memory block, which stores additional information from the entire defogging training set. The reference frame features are used as queries to perform query operations in the global memory block. According to the query results, matching additional information is further selected from the memory block to enhance the reference frame features. Finally, two learnable parameters are used to fuse the features of the two.

Images