Video defogging method and system based on multi-scale space-time fusion network

Abstract

The invention relates to the technical field of computer vision, and particularly discloses a video defogging method and system based on a multi-scale space-time fusion network, and the method comprises the steps: generating a video sequence image through framing processing, and carrying out the preprocessing through an automatic color equalization technology, so as to optimize a visual effect; an encoder based on a channel attention mechanism is adopted, color recovery and layer-by-layer feature extraction of the image are achieved from five dimensions, and low-frequency information in the image is effectively reserved; low-frequency information in the shallow features is further transmitted through a low-frequency information transmission module, and feature response is dynamically adjusted to improve the utilization efficiency of the shallow features; space-time feature alignment and fusion are carried out through a decoder, so that the capturing capability of the model on time sequence and space information is enhanced; and the fused high-dimensional features are converted into an image in a target RGB format through a reconstruction module, so that the spatial resolution and the color precision of the output image are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video defogging method and system based on a multi-scale spatiotemporal fusion network. Background Art

[0002] Dehazing algorithm is an important research direction in the field of computer vision. It aims to restore and enhance haze images to make them clearer and more realistic, and effectively deal with the interference caused by bad weather and environment. With the development of intelligent driving and smart transportation, video data with haze will have a great impact on downstream tasks, including target detection, image segmentation, etc. In order to reduce the resulting video degradation, video data must be dehazed. The haze phenomenon in the atmosphere is caused by the scattering of light by tiny particles suspended in the atmosphere. This optical effect will cause visual degradation of images and videos.

[0003] Due to the complex interplay of scattering and absorption phenomena triggered by tiny atmospheric particles, the perceptual sensitivity of visual content is compromised. As a result, the resulting images suffer from reduced contrast and white haze coverage. Such images, characterized by haze, undergo substantial degradation, leading to loss of critical information content. Strategic deployment of dehazing techniques emerges as a viable approach to correct the haze-induced degradations that are prevalent in images and frameworks.

[0004] Video dehazing benefits from temporal cues, such as highly correlated haze thickness and lighting conditions as well as moving foreground objects and backgrounds. However, existing video dehazing methods have some limitations. They obtain haze-free frames from component estimation based on physical models, and the intermediate predictions are inaccurate, which leads to the accumulation of errors in the final results. Secondly, these methods aggregate temporal information using input feature superposition or frame-to-frame alignment in local sliding windows, which makes it difficult to obtain global and long-term temporal information. Summary of the invention

[0005] The present invention provides a video defogging method and system based on a multi-scale spatiotemporal fusion network, and solves the technical problem of how to perform high-quality defogging on video images.

[0006] In order to solve the above technical problems, the present invention provides a video defogging method based on a multi-scale spatiotemporal fusion network, comprising the steps of:

[0007] S1, performing frame processing on the haze video to be processed to generate an original image frame sequence consisting of multiple original image frames in time sequence;

[0008] S2, preprocessing the original image frame sequence using automatic color balancing technology to obtain a corresponding preprocessed image frame sequence;

[0009] S3, inputting the original image frame sequence and the pre-processed image frame sequence into the constructed multi-scale spatiotemporal fusion network for defogging, to obtain a defogging image frame sequence;

[0010] The multi-scale spatiotemporal fusion network includes an encoder, a low-frequency information transmission module, a decoder and a reconstruction module; the encoder module is used to take the current frame, the three frames before the current frame, a total of four frames of pre-processed image frames and original image frames as input, perform feature extraction from five different dimensions, and obtain feature enhanced image frames of five dimensions; the low-frequency information transmission module is used to extract low-frequency information from the feature enhanced image frames of five dimensions, and obtain the corresponding low-frequency information of five dimensions to input into the decoder; the decoder is used to take the feature enhanced image frames of the fifth dimension and the low-frequency information of five dimensions as input, perform spatiotemporal feature shift and spatiotemporal feature fusion, and obtain feature fused image frames; the reconstruction module is used to reconstruct the feature fused image frames to obtain corresponding four frames of fog-free image frames.

[0011] Furthermore, the encoder is sequentially provided with five feature extraction units, and the processing flow of the encoder comprises the steps of:

[0012] B1, inputting four pre-processed image frames and original image frames into a first feature extraction unit to extract features of a first dimension, thereby obtaining four feature-enhanced image frames of the first dimension;

[0013] B2, inputting the four feature-enhanced image frames of the first dimension and the four original image frames into a second feature extraction unit to perform feature extraction of the second dimension, thereby obtaining four feature-enhanced image frames of the second dimension;

[0014] B3, inputting the four feature-enhanced image frames of the second dimension and the four original image frames into a third feature extraction unit to perform feature extraction of the third dimension, thereby obtaining four feature-enhanced image frames of the third dimension;

[0015] B4, inputting the four feature-enhanced image frames of the third dimension and the four original image frames into a fourth feature extraction unit to perform feature extraction of the fourth dimension, thereby obtaining four feature-enhanced image frames of the fourth dimension;

[0016] B5, inputting the four feature-enhanced image frames of the fourth dimension and the four original image frames into a fifth feature extraction unit to perform feature extraction of the fifth dimension, thereby obtaining four feature-enhanced image frames of the fifth dimension;

[0017] B6. Input four feature-enhanced image frames in five dimensions into the low-frequency information transmission module, and input four feature-enhanced image frames in the fifth dimension into the encoder.

[0018] Furthermore, the feature extraction process of each feature extraction unit includes the steps of:

[0019] G1, convolve the input feature map with the input feature map F G0 Perform channel connection to obtain the feature map F G1 ;

[0020] G2, feature map F G1 Convolution, PReLU activation function, and convolution are performed successively to generate feature map F G2 ;

[0021] G3, feature map F G2 Perform global average pooling, convolution, PReLU activation function, convolution, and Sigmoid function to generate weight information; combine the weight information with the feature map F G2 Multiply to get the feature map F G3 ;

[0022] G4, input feature map F G0 With the feature map F G3 Add together to get the feature map F G4 ;

[0023] G5, feature map F G4 Convolution, convolution, and convolution are performed successively to obtain the feature map F G5 .

[0024] Further, the low-frequency information transmission module includes four low-frequency information transmission units, which are respectively used to extract low-frequency information of the feature-enhanced image frames output by the second to fifth feature extraction units;

[0025] The operation of each of the low frequency information transmission units comprises the steps of:

[0026] L1, extract shallow features: extract initial low-frequency features from the feature extraction unit;

[0027] L2, transfer shallow features: use long-distance skip connections to pass the initial low-frequency features directly to the decoder.

[0028] Furthermore, in the transfer process of step L2, a multi-scale mapping unit is introduced to process the shallow features. The operation performed by the multi-scale mapping unit includes the following steps:

[0029] L21, through four convolution operations with different convolution kernel sizes, four sets of feature maps of different scales are generated for the input feature map;

[0030] L22, through the element-by-element sum operation, the four sets of feature maps of different scales are fused to obtain a fused feature map;

[0031] L23, extract the global information of the fused feature map through global average pooling;

[0032] L24, pass the global information through the fully connected layer to obtain compact features;

[0033] L25, pass the compact features through 4 fully connected layers to obtain the weights corresponding to each scale;

[0034] L26, perform weighted summation on the weight and the input feature map of the corresponding scale to obtain the final output feature map;

[0035] After the multi-scale mapping unit, there is also a step of:

[0036] L27. Reconstruct the output feature map of the multi-scale mapping unit to generate a high-quality reconstructed feature map and input it into the decoder.

[0037] Furthermore, the decoder includes four spatiotemporal feature shifting and fusion units, the feature enhanced image frame of the fifth feature extraction unit and the reconstructed feature map output by the fourth low-frequency information transmission unit are input into the first spatiotemporal feature shifting and fusion unit for the first spatiotemporal feature shifting and fusion to obtain a first fused feature map; the first fused feature map and the reconstructed feature map output by the third low-frequency information transmission unit are input into the second spatiotemporal feature shifting and fusion unit for the second spatiotemporal feature shifting and fusion to obtain a second fused feature map; the second fused feature map and the reconstructed feature map output by the second low-frequency information transmission unit are input into the third spatiotemporal feature shifting and fusion unit for the third spatiotemporal feature shifting and fusion to obtain a third fused feature map; the third fused feature map and the reconstructed feature map output by the first low-frequency information transmission unit are input into the fourth spatiotemporal feature shifting and fusion unit for the fourth spatiotemporal feature shifting and fusion to obtain a fourth fused feature map, which is input into the reconstruction module.

[0038] Furthermore, the operation of each spatiotemporal feature shift and fusion unit specifically includes the following steps:

[0039] T1, aggregate current frame f i The previous frame f i-1 , aggregate the current frame f i The two previous frames f i-1 、f i-2 , aggregate the current frame f i The first three frames f i-1 、f i-2 、f i-3 , and obtain three groups of aggregate features;

[0040] T2, using three spatiotemporal shift modules to perform spatiotemporal shift on the three groups of aggregate features respectively, to obtain three groups of shifted aggregate features;

[0041] T3, fuse the three groups of shifted aggregation features to obtain fused aggregation features;

[0042] T4. Upsample the fused aggregated features to obtain high-dimensional features and input them into the reconstruction module.

[0043] Furthermore, the operation performed by the space-time shift module includes the steps of:

[0044] T21, spatially shift a set of input aggregate features to obtain spatially shifted features;

[0045] T22, apply the window multi-head self-attention mechanism to the spatial shift features and the current frame f i A part of each spatial shift feature generates multiple local windows, and generates a corresponding K value matrix and V value matrix for each local window of the spatial shift feature. i The local window generates the Q value matrix;

[0046] T23. Fuse the K value matrix, V value matrix and Q value matrix of each local window to obtain the local window fusion feature; fuse all local window fusion features to obtain the shifted aggregation feature corresponding to the group of aggregation features.

[0047] Further, for the current frame f i The previous frame f i-1 The first group of aggregated features, step T21 specifically includes the steps of:

[0048] T211, the current frame f i Divided into f i a and f i b Two parts, the previous frame f i-1 Divided into and Two parts;

[0049] T212, group the features Slice along the channel dimension to obtain M feature slices, where the mth feature slice is expressed as Where m=1,....,M is the slice index;

[0050] T213. For each feature slice Shift it spatially in the x and y directions to obtain the translated feature piece Group all features along the channel dimension Connect in series to get the spatial displacement characteristics

[0051] T214, f i b and Carry out series connection, and compare the series connection results and spatial displacement characteristics After convolution, add the concatenated result, convolve the added result and add the added result again to obtain a first feature map;

[0052] T215, yes After convolution and Add, convolve the added result and add it again to obtain the second feature map;

[0053] T216, connecting the first feature map and the second feature map in series to obtain spatial shift features of a first group of aggregated features;

[0054] T217, f i a As the current frame f i Multi-head self-attention mechanism for a part of the input window.

[0055] The present invention also provides a video defogging system based on a multi-scale spatiotemporal fusion network, the key of which is that it is provided with an intelligent agent, which is used to execute the video defogging method based on the multi-scale spatiotemporal fusion network.

[0056] The video defogging method and system based on the multi-scale spatiotemporal fusion network provided by the present invention generates video sequence images through frame processing, and applies automatic color equalization technology for preprocessing to optimize the visual effect; an encoder based on the channel attention mechanism is used to realize color restoration and layer-by-layer feature extraction of the image from five dimensions respectively, effectively retaining the low-frequency information in the image; the low-frequency information in the shallow features is further transmitted through the low-frequency information transmission module, and the feature response is dynamically adjusted to improve the utilization efficiency of the shallow features; the spatiotemporal feature alignment and fusion are performed through the decoder, thereby enhancing the model's ability to capture temporal and spatial information; the fused high-dimensional features are converted into the target RGB format image through the reconstruction module, ensuring the spatial resolution and color accuracy of the output image. On the whole, the present invention and system provide an efficient and accurate video defogging technology, which is suitable for downstream application scenarios such as smart transportation and smart driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is an overall structural diagram based on a multi-scale spatiotemporal fusion network provided by an embodiment of the present invention;

[0058] Figure 2 is a structural diagram of a first feature extraction unit (GFEB1) provided in an embodiment of the present invention;

[0059] Figure 3 is a structural diagram of a spatiotemporal feature shift and fusion unit (STAF) provided in an embodiment of the present invention;

[0060] Figure 4It is a structural diagram of a spatiotemporal shift module (STFS) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0062] The video defogging method based on a multi-scale spatiotemporal fusion network provided by an embodiment of the present invention comprises the following steps:

[0063] S1, performing frame processing on the haze video to be processed to generate an original image frame sequence consisting of multiple original image frames in time sequence;

[0064] S2, preprocessing the original image frame sequence using automatic color balancing technology to obtain a corresponding preprocessed image frame sequence;

[0065] S3. Input the original image frame sequence and the preprocessed image frame sequence into the constructed multi-scale spatiotemporal fusion network for defogging to obtain a defogging image frame sequence.

[0066] (1) Step S1

[0067] Each frame of the haze video is extracted independently to provide basic data for subsequent processing steps.

[0068] (2) Step S2

[0069] Step S2 applies the automatic color equalization (ACE) technology to each original image frame to improve the overall visual effect by adjusting the brightness and contrast of the image and performing preliminary color correction. Then, the image after color equalization is saved as one of the inputs of the multi-scale spatiotemporal fusion network to provide optimized image data for subsequent processing steps.

[0070] Specifically, for each original image frame, the automatic color balancing technology specifically includes the following steps:

[0071] S21, Color Space Adjustment: Adjust the brightness, contrast and color balance of the image to ensure that the image is more visually balanced and natural, complete the color difference correction of the image, which can significantly improve the visibility of the image and make it more suitable for subsequent processing;

[0072] S22, dynamic expansion: Dynamically expand the corrected image to improve the contrast and brightness of the image. The dynamic expansion formula is as follows:

[0073]

[0074] Wherein, R(x) represents the pixel value of the pixel point x in the original image frame obtained in step S21, L(x) represents the pixel value of the pixel point x obtained by expansion, and R min represents the minimum value of all pixel values ​​in the entire original image frame after step S21, R max represents the maximum value of all pixel values ​​in the entire original image frame after step S21. By subtracting the minimum value and dividing by the range, R(x) is converted to the interval [0,1], so that the contrast of the image is enhanced.

[0075] Through the above steps, ACE generates images with high contrast and enhanced brightness. ACE can be regarded as a simplified model of the human visual system. Its enhancement process is consistent with human perception, which helps to generate high-quality images that are more in line with visual perception during video processing.

[0076] (3) Step S3

[0077] 1) The overall structure of the multi-scale spatiotemporal fusion network

[0078] The structure of the multi-scale spatiotemporal fusion network is as follows: Figure 1 As shown, it includes an encoder, a low-frequency information transmission module, a decoder and a reconstruction module. The encoder module is used to take the current frame, the three frames before the current frame, a total of four pre-processed image frames and the original image frames as input, extract features from 5 different dimensions, and obtain feature enhanced image frames of five dimensions. The low-frequency information transmission module is used to extract low-frequency information from the feature enhanced image frames of five dimensions, and obtain the corresponding low-frequency information of five dimensions to input into the decoder. The decoder is used to take the feature enhanced image frames of the fifth dimension and the low-frequency information of five dimensions as input, perform spatiotemporal feature shift and spatiotemporal feature fusion, and obtain feature fused image frames; the reconstruction module is used to reconstruct the feature fused image frames to obtain the corresponding four frames of fog-free image frames.

[0079] Among them, the encoder realizes color restoration and layer-by-layer feature extraction of the image from five dimensions based on the channel attention mechanism, effectively retaining the low-frequency information in the image. Afterwards, the low-frequency information transmission module further transmits the low-frequency information in the shallow features through long-distance jump connections and multi-scale mapping units, and dynamically adjusts the feature response to improve the utilization efficiency of shallow features. The decoder enhances the model's ability to capture temporal and spatial information through grouped spatial displacement and window multi-head self-attention mechanism. Finally, the fused high-dimensional features are converted into the target RGB format image through the reconstruction module to ensure the spatial resolution and color accuracy of the output image.

[0080] 2) Encoder

[0081] The structure of the encoder is as follows Figure 1As shown, five feature extraction units (CFEB1 to CFEB5) are sequentially provided, and the processing flow of the encoder includes the steps:

[0082] B1, inputting four pre-processed image frames and original image frames into the first feature extraction unit (GFEB1) to extract features of the first dimension, and obtaining four feature-enhanced image frames of the first dimension;

[0083] B2, inputting the four feature-enhanced image frames of the first dimension and the four original image frames into a second feature extraction unit (GFEB2) to extract features of the second dimension, thereby obtaining four feature-enhanced image frames of the second dimension;

[0084] B3, inputting the four feature-enhanced image frames of the second dimension and the four original image frames into a third feature extraction unit (GFEB3) to extract features of the third dimension, thereby obtaining four feature-enhanced image frames of the third dimension;

[0085] B4, inputting the four feature-enhanced image frames of the third dimension and the four original image frames into a fourth feature extraction unit (GFEB4) to perform feature extraction of the fourth dimension, thereby obtaining four feature-enhanced image frames of the fourth dimension;

[0086] B5, inputting the four feature-enhanced image frames of the fourth dimension and the four original image frames into a fifth feature extraction unit (GFEB5) to perform feature extraction of the fifth dimension, thereby obtaining four feature-enhanced image frames of the fifth dimension;

[0087] B6. Input the four feature-enhanced image frames in five dimensions into the low-frequency information transmission module, and input the four feature-enhanced image frames in the fifth dimension into the encoder.

[0088] As an example, the input and output dimensions of the five feature extraction units are as follows:

[0089] GFEB1: The initial input dimension is 3 and the final output dimension is 64;

[0090] GFEB2: The initial input dimension is 64+12, and the final output dimension is 96;

[0091] GFEB3: The initial input dimension is 96+24, and the final output dimension is 192;

[0092] GFEB4: the initial input dimension is 192+48, and the final output dimension is 384;

[0093] GFEB5: The initial input dimension is 384+96, and the final output dimension is 768.

[0094] The feature extraction process of each GFEB unit is similar. Taking GFEB1 as an example, refer to Figure 2The structural diagram of GFEB1 shown in FIG. 1 , the feature extraction process of GFEB1 includes the following steps:

[0095] G1, convolve the input feature map with the input feature map F G0 Perform channel connection (Cat operation) to obtain the feature map F G1 ;

[0096] G2, feature map F G1 Convolution (conv, input dimension: 3, output dimension: 3, convolution kernel: 3×3, stride: 1), PReLU activation function (nonlinear transformation of the feature map after convolution to improve the expression ability of the model), convolution (conv, input dimension: 3, output dimension: 3, convolution kernel: 3×3, stride: 1) are performed successively to generate the feature map F G2 ;

[0097] G3, feature map F G2 Perform global average pooling (extract global feature information and provide context information for the attention mechanism), convolution (input dimension: 3, output dimension: 1, convolution kernel: 1×1, stride: 1), PReLU activation function (process the channel feature map to maintain nonlinearity), convolution (input dimension: 1, output dimension: 3, convolution kernel: 1×1, stride: 1, provide finer feature adjustment), Sigmoid function (generate attention weight map for adaptive adjustment of feature weights of each channel), generate weight information; combine the weight information with the feature map F G2 Multiply to get the feature map F G3 ;

[0098] G4, input feature map F G0 With the feature map F G3 Add (reduce the attenuation of low-frequency information and retain more image details) to obtain the feature map F G4 ;

[0099] G5, feature map F G4 Convolution (input dimension: 3, output dimension: 64, convolution kernel: 3×3, stride: 1), convolution (convolution operation on input feature map, input dimension: 64, output dimension: 64, convolution kernel: 3×3, stride: 2), convolution (input dimension: 64, output dimension: 64, convolution kernel: 3×3, stride: 1) are performed successively to obtain the feature map F G5 .

[0100] 3) Low-frequency information transmission module

[0101] After obtaining the high-quality feature map processed by the encoder, in order to retain the low-frequency information in the shallow features extracted by the encoder and ensure that this information can be efficiently used, this embodiment directly passes the low-frequency information to the decoder through a long-distance jump connection, and introduces a multi-scale mapping unit in this process.

[0102] like Figure 1 As shown, the low-frequency information transfer module includes four low-frequency information transfer units LFITM1 to LFITM4, which are respectively used to extract low-frequency information of the feature enhanced image frames output by GFEB2 to GFEB5.

[0103] The structure of each low-frequency information transmission unit is the same. The operation of any low-frequency information transmission unit includes the steps of:

[0104] L1, extract shallow features: extract initial low-frequency features from the feature extraction unit;

[0105] L2, transfer shallow features: Use long-distance skip connections to pass the initial low-frequency features directly to the decoder, which prevents these shallow features from being lost in the deep network and retains more image details.

[0106] During the transfer process of step L2, a multi-scale mapping unit is introduced to process the shallow features so as to better integrate them with the deep features extracted by the decoder.

[0107] Multi-scale mapping unit: Starting from the multi-scale, multiple mapping layers with different receptive field sizes are designed, and global information representation is obtained based on them. Furthermore, the softmax attention guided by different scale information is used to obtain the weight representation corresponding to each scale, and finally they are weighted summed. The specific steps include:

[0108] L21, multi-scale convolution operation: through four convolution operations with different convolution kernel sizes (the convolution kernels are 1×1, 3×3, 5×5, and 7×7 respectively), four sets of feature maps of different scales are generated for the input feature map;

[0109] L22, multi-scale feature fusion: through the element-by-element summation operation, four sets of feature maps of different scales are fused to obtain a fused feature map;

[0110] L23, global average pooling: extract the global information of the fused feature map;

[0111] L24, fully connected layer: The global information is passed through the fully connected layer to obtain compact features, thereby achieving more accurate and adaptive guidance;

[0112] L25, four fully connected layers: the compact features are passed through four fully connected layers to obtain the weights corresponding to each scale;

[0113] L26, feature weighting: The weight is added to the input feature map of the corresponding scale to obtain the final output feature map.

[0114] After the multi-scale mapping unit, there are also steps:

[0115] L27, feature reconstruction: Reconstruct the output feature map of the multi-scale mapping unit to generate a high-quality reconstructed feature map input into the decoder.

[0116] The low-frequency information transmission module can effectively retain and utilize the low-frequency information in shallow features, fully integrate all extracted high- and low-dimensional features, improve the overall feature expression capability, and ensure the stability of the training process.

[0117] 4) Decoder

[0118] like Figure 2 As shown, the decoder includes four spatiotemporal feature shifting and fusion units STAF1 and STAF4, the feature enhanced image frame of the fifth feature extraction unit and the reconstructed feature map output by the fourth low-frequency information transmission unit are input into the first spatiotemporal feature shifting and fusion unit STAF1 for the first spatiotemporal feature shifting and fusion to obtain a first fused feature map; the first fused feature map and the reconstructed feature map output by the third low-frequency information transmission unit are input into the second spatiotemporal feature shifting and fusion unit STAF2 for the second spatiotemporal feature shifting and fusion to obtain a second fused feature map; the second fused feature map and the reconstructed feature map output by the second low-frequency information transmission unit are input into the third spatiotemporal feature shifting and fusion unit STAF3 for the third spatiotemporal feature shifting and fusion to obtain a third fused feature map; the third fused feature map and the reconstructed feature map output by the first low-frequency information transmission unit are input into the fourth spatiotemporal feature shifting and fusion unit STAF4 for the fourth spatiotemporal feature shifting and fusion to obtain a fourth fused feature map input into the reconstruction module.

[0119] The operation of each spatiotemporal feature shift and fusion unit is the same, refer to Figure 3 The structure diagram of the spatiotemporal feature shift and fusion unit shown in FIG. 1 , the operation of each spatiotemporal feature shift and fusion unit specifically includes the following steps:

[0120] T1, aggregate current frame f i The previous frame f i-1 , aggregate the current frame f i The two previous frames f i-1 、f i-2 , aggregate the current frame f i The first three frames f i-1 、f i-2 、f i-3 , and obtain three groups of aggregate features;

[0121] T2, using three spatiotemporal shift modules (STFS) to perform spatiotemporal shift on the three groups of aggregated features respectively, and obtain three groups of shifted aggregated features (so that the model can better capture spatial information and align features);

[0122] T3, fuse the three groups of shifted aggregation features to obtain fused aggregation features;

[0123] T4. Upsample the fused aggregated features to obtain high-dimensional features and input them into the reconstruction module.

[0124] Among them, reference Figure 4 The structure diagram of the spatiotemporal shift module shown in FIG. 1 , the operation performed by the spatiotemporal shift module (STFS) includes the following steps:

[0125] T21, spatially shift a set of input aggregate features to obtain spatially shifted features;

[0126] T22, apply the window multi-head self-attention mechanism (W-MSA) to the spatial shift features and the current frame f i A part of each spatial shift feature generates multiple local windows, and generates a corresponding K (Key) value matrix and V (Value) value matrix for each local window of the spatial shift feature. i The local window generates a Q (Query) value matrix;

[0127] T23. Fuse the K value matrix, V value matrix and Q value matrix of each local window to obtain the local window fusion feature; fuse all local window fusion features to obtain the shifted aggregation feature corresponding to the group of aggregation features (further enhance the expression ability of the features so that the model can better capture and utilize spatial information).

[0128] like Figure 4 As shown, for the current frame f i The previous frame f i-1 The first group of aggregated features, step T21 specifically includes the steps of:

[0129] T211, the current frame f i Divided into f i a and f i b Two parts, the previous frame f i-1 Divided into and Two parts;

[0130] T212, Feature Slicing: Feature Groups M feature slices are obtained along the channel dimension, and the mth feature slice is expressed as Where m=1,....,M is the slice index;

[0131] T213, translation feature slice: for each feature slice Apply Δx to it in the x and y directions m ,Δy m ∈{-9,-5,0,5,9} pixels of spatial shift (Spatial Shift) to obtain the translated feature slice

[0132]

[0133] |Δx m |=k x *(s-1)+1,|Δy m |=k y *(s-1)+1

[0134] Among them, Shift() represents the shift operation, k x , k y is an integer, and s is defined as the base length of the spatial shift. When the spatial shift results in a blank pixel in the boundary, we set s to zero. m Pixel displacement, the corresponding feature group is spatially shifted by Δx m -1 pixel displacement, followed by a depthwise 3×3 convolution that spans two displacements and achieves a smooth translation between two adjacent displaced feature slices. All feature groups are then grouped together along the channel dimension. Concat them together to get the spatial displacement feature

[0135]

[0136] T214, f i b and Concat the concatenation results and spatial displacement features After convolution, add the concatenated result, convolve the added result and add the added result again to obtain a first feature map;

[0137] T215, yes After convolution and Add, convolve the added result and add it again to obtain the second feature map;

[0138] T216, connecting the first feature map and the second feature map in series to obtain spatial shift features of a first group of aggregated features;

[0139] T217, f i a As the current frame f i A part of the input window multi-head self-attention mechanism (W-MSA).

[0140] For the current frame f i The two previous frames f i-1 、f i-2 The second set of aggregated features, step T21 is similar to the above steps T211 to T216, in that the generated second feature map adds the i-2 For the spatial shift feature including the current frame f i The first three frames f i-1 、f i-2 、f i-3 The second set of aggregated features, step T21 is similar to the above steps T211 to T216, in that the generated second feature map adds the i-2 The spatial shift feature of f is then generated. i-3 The third feature map of the spatial shift feature is obtained, and finally the first feature map, the second feature map and the third feature map are connected in series.

[0141] The window multi-head self-attention mechanism (W-MSA) specifically includes the following steps:

[0142] T221. Window division: The feature map is divided into multiple local windows of the same size, each window containing a specific number of pixel blocks.

[0143] T222, Multi-head self-attention mechanism: In each local window, the query, key, and value matrices are calculated separately. The multi-head self-attention mechanism is used to calculate the attention weight of each position and perform weighted summation on the features. This mechanism enables the features at each position to interact with other positions in the window, thereby enhancing the feature expression capability.

[0144] T223, fusion features: fuse the features in each window to generate a new feature map, further improving the expressiveness of the features and enabling the model to better capture and utilize spatial information.

[0145] Step T3 is as follows: stack the three sets of features after spatiotemporal shift, use the attention mechanism, and improve the understanding and processing capabilities of temporal features by modeling the dependencies in the time dimension. The result of the above multiplication is normalized by Softmax, and then the processed features are aggregated with the features of the current frame. By fusing information from different time points, a more comprehensive feature representation is finally generated. An upsampling structure is used between adjacent STFS modules, and PixelShufflePack is used to shuffle the pixels of the channels to reduce the dimension while improving the resolution.

[0146] More specifically, for the first STFS module, step T3 includes the steps of:

[0147] T31, spatiotemporal feature stacking: Stack the three sets of features after STFS to form a feature group containing multiple time dimension information. These feature groups represent image information at different time points to capture subtle changes in the time dimension. Assuming that the three sets of features are F1, F2, and F3, the stacked features can be expressed as:

[0148] F stack =Stack(F1,F2,F3)

[0149] The stacked feature F stack The dimensions are (N, 3, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and width respectively.

[0150] T32, query feature generation: extract the query feature F of the current frame query , and flatten it into a two-dimensional matrix Q:

[0151] Q = Flatten (F query )

[0152] T33, correlation calculation: Compare the two-dimensional matrix Q of the query feature with the stacked feature F stack Perform matrix multiplication to calculate the correlation between the current frame and the features at different times. Stacked features F stack are converted into two-dimensional matrices K and V (via a flattening operation):

[0153] K = Flatten (F stack )

[0154] V=Flatten(F stack )

[0155] Then, the correlation between the query feature Q and the key K is calculated:

[0156]

[0157] Among them, the dimension of A is (N×H×W,3), which represents the correlation between features at different time points, and the superscript T represents the matrix transpose.

[0158] This step models the dependencies in the time dimension through the attention mechanism, so that the model can better understand and process temporal features.

[0159] T34, attention weighting and normalization: The above correlation result A is normalized through the Softmax function:

[0160] α=Softmax(A)

[0161] Among them, α is the weight assigned to the features at different time points. The Softmax function can assign weights to the features at different time points, so that the features with higher correlation receive more attention, thereby enhancing the expressiveness of the features.

[0162] T35, feature aggregation: Finally, the values ​​are weighted and summed using the attention weight α to obtain the aggregated features:

[0163] F out =α·V

[0164] Finally, the aggregated feature F out With the feature F of the current frame query To perform the fusion:

[0165] F final =Concat(F out ,F query )

[0166] Finally, it passes through a convolutional layer to further process and optimize the final feature representation.

[0167] T36, upsampling: Upsample the feature map generated in step T35 through convolution and pixel shuffling, reducing the dimension while improving the resolution. The specific process is as follows:

[0168] T361, convolution layer: performs convolution operation on the input feature map, expands the number of channels, and prepares for subsequent pixel shuffling.

[0169] T362, Pixel Shuffle: Perform pixel shuffle on the feature map after convolution. Pixel shuffle improves the spatial resolution of the feature map and rearranges the increased number of channels to the spatial dimension.

[0170] T37, return the upsampled feature map, and perform the above operation for the next STFS module.

[0171] 5) Reconstruction module

[0172] The reconstruction module first performs the first convolution operation on the input feature map, and then applies a nonlinear activation function to extract and enhance the salient features within it. Then, through further convolution operations, the number of channels of the feature map is gradually reduced from a higher dimension to 3 channels of the target output to ensure that the final result can match the RGB format of the target image. Throughout the process, the spatial resolution of the feature map is maintained, ensuring that the features after nonlinear transformation can be effectively extracted and integrated, providing a solid foundation for accurate image reconstruction.

[0173] The specific process of rebuilding the module is as follows:

[0174] R1, convolution layer: perform convolution operation on the input feature map, convolution kernel: 3×3, stride: 1, keep the spatial size of the output image unchanged.

[0175] R2, LeakyReLU activation function: The output of the first layer of convolution is activated using the LeakyReLU function, with a negative slope parameter of 0.1, so that when the input is negative, the output is 0.1 times the value.

[0176] R3, convolution layer: After receiving the LeakyReLU activation, it is processed using a 1×1 convolution kernel to reduce the number of channels to 3 channels of the target output. Among them, the stride is 1 and the padding is 0, keeping the spatial size of the output image unchanged.

[0177] In order to apply the above method, an embodiment of the present invention also provides a video defogging system based on a multi-scale spatiotemporal fusion network, which is provided with an intelligent agent, which is used to execute the above video defogging method based on a multi-scale spatiotemporal fusion network.

[0178] In summary, the embodiments of the present invention provide a video defogging method and system based on a multi-scale spatiotemporal fusion network, which generates video sequence images through frame processing, and applies automatic color equalization technology for preprocessing to optimize the visual effect; an encoder based on a channel attention mechanism is used to realize color restoration and layer-by-layer feature extraction of the image from five dimensions, respectively, effectively retaining the low-frequency information in the image; the low-frequency information in the shallow features is further transmitted through the low-frequency information transmission module, and the feature response is dynamically adjusted to improve the utilization efficiency of the shallow features; the spatiotemporal feature alignment and fusion are performed through the decoder, which enhances the model's ability to capture temporal and spatial information; the fused high-dimensional features are converted into the target RGB format image through the reconstruction module, ensuring the spatial resolution and color accuracy of the output image. On the whole, the present invention and system provide an efficient and accurate video defogging technology, which is suitable for downstream application scenarios such as smart transportation and smart driving.

[0179] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A video dehazing method based on a multi-scale spatiotemporal fusion network, characterized in that: Includes steps: S1, performing frame processing on the haze video to be processed to generate an original image frame sequence consisting of multiple original image frames in time sequence; S2, preprocessing the original image frame sequence using automatic color balancing technology to obtain a corresponding preprocessed image frame sequence; S3, inputting the original image frame sequence and the pre-processed image frame sequence into the constructed multi-scale spatiotemporal fusion network for defogging, to obtain a defogging image frame sequence; The multi-scale spatiotemporal fusion network includes an encoder, a low-frequency information transmission module, a decoder and a reconstruction module; the encoder module is used to take the current frame, the three frames before the current frame, a total of four frames of pre-processed image frames and original image frames as input, perform feature extraction from five different dimensions, and obtain feature enhanced image frames of five dimensions; the low-frequency information transmission module is used to extract low-frequency information from the feature enhanced image frames of five dimensions, and obtain the corresponding low-frequency information of five dimensions to input into the decoder; the decoder is used to take the feature enhanced image frames of the fifth dimension and the low-frequency information of five dimensions as input, perform spatiotemporal feature shift and spatiotemporal feature fusion, and obtain feature fused image frames; the reconstruction module is used to reconstruct the feature fused image frames to obtain corresponding four frames of fog-free image frames.

2. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 1 is characterized in that: The encoder is provided with five feature extraction units in sequence, and the processing flow of the encoder comprises the steps of: B1, inputting four pre-processed image frames and original image frames into a first feature extraction unit to extract features of a first dimension, thereby obtaining four feature-enhanced image frames of the first dimension; B2, inputting the four feature-enhanced image frames of the first dimension and the four original image frames into a second feature extraction unit to perform feature extraction of the second dimension, thereby obtaining four feature-enhanced image frames of the second dimension; B3, inputting the four feature-enhanced image frames of the second dimension and the four original image frames into a third feature extraction unit to perform feature extraction of the third dimension, thereby obtaining four feature-enhanced image frames of the third dimension; B4, inputting the four feature-enhanced image frames of the third dimension and the four original image frames into a fourth feature extraction unit to perform feature extraction of the fourth dimension, thereby obtaining four feature-enhanced image frames of the fourth dimension; B5, inputting the four feature-enhanced image frames of the fourth dimension and the four original image frames into a fifth feature extraction unit to perform feature extraction of the fifth dimension, thereby obtaining four feature-enhanced image frames of the fifth dimension; B6. Input four feature-enhanced image frames in five dimensions into the low-frequency information transmission module, and input four feature-enhanced image frames in the fifth dimension into the encoder.

3. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 2 is characterized in that: The feature extraction process of each feature extraction unit includes the following steps: G1, convolve the input feature map with the input feature map F G0 Perform channel connection to obtain the feature map F G1 ; G2, feature map F G1 Convolution, PReLU activation function, and convolution are performed successively to generate feature map F G2 ; G3, feature map F G2 Perform global average pooling, convolution, PReLU activation function, convolution, and Sigmoid function in sequence to generate weight information; combine the weight information with the feature map F G2 Multiply to get the feature map F G3 ; G4, input feature map F G0 With the feature map F G3 Add together to get the feature map F G4 ; G5, feature map F G4 Convolution, convolution, and convolution are performed successively to obtain the feature map F G5 .

4. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 3 is characterized in that: The low-frequency information transmission module includes four low-frequency information transmission units, which are respectively used to extract the low-frequency information of the feature-enhanced image frames output by the second to fifth feature extraction units; The operation of each of the low frequency information transmission units comprises the steps of: L1, extract shallow features: Extracting initial low-frequency features from a feature extraction unit; L2, transfer shallow features: use long-distance skip connections to pass the initial low-frequency features directly to the decoder.

5. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 4 is characterized in that: In the transfer process of step L2, a multi-scale mapping unit is introduced to process the shallow features. The operations performed by the multi-scale mapping unit include the following steps: L21, through four convolution operations with different convolution kernel sizes, four sets of feature maps of different scales are generated for the input feature map; L22, through the element-by-element sum operation, the four sets of feature maps of different scales are fused to obtain a fused feature map; L23, extract the global information of the fused feature map through global average pooling; L24, pass the global information through the fully connected layer to obtain compact features; L25, pass the compact features through 4 fully connected layers to obtain the weights corresponding to each scale; L26, perform weighted summation on the weight and the input feature map of the corresponding scale to obtain the final output feature map; After the multi-scale mapping unit, there is also a step of: L27. Reconstruct the output feature map of the multi-scale mapping unit to generate a high-quality reconstructed feature map and input it into the decoder.

6. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 5 is characterized in that: The decoder includes four spatiotemporal feature shifting and fusion units. The feature enhanced image frame of the fifth feature extraction unit and the reconstructed feature map output by the fourth low-frequency information transmission unit are input into the first spatiotemporal feature shifting and fusion unit for the first spatiotemporal feature shifting and fusion to obtain a first fused feature map; the first fused feature map and the reconstructed feature map output by the third low-frequency information transmission unit are input into the second spatiotemporal feature shifting and fusion unit for the second spatiotemporal feature shifting and fusion to obtain a second fused feature map; the second fused feature map and the reconstructed feature map output by the second low-frequency information transmission unit are input into the third spatiotemporal feature shifting and fusion unit for the third spatiotemporal feature shifting and fusion to obtain a third fused feature map; the third fused feature map and the reconstructed feature map output by the first low-frequency information transmission unit are input into the fourth spatiotemporal feature shifting and fusion unit for the fourth spatiotemporal feature shifting and fusion to obtain a fourth fused feature map and input into the reconstruction module.

7. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 6 is characterized in that: The operation of each spatiotemporal feature shift and fusion unit specifically includes the following steps: T1, aggregate current frame f i The previous frame f i-1 , aggregate the current frame f i The two previous frames f i-1 、f i-2 , aggregate the current frame f i The first three frames f i-1 、f i-2 、f i-3 , and obtain three groups of aggregate features; T2, using three spatiotemporal shift modules to perform spatiotemporal shift on the three groups of aggregate features respectively, to obtain three groups of shifted aggregate features; T3, fuse the three groups of shifted aggregation features to obtain fused aggregation features; T4. Upsample the fused aggregated features to obtain high-dimensional features and input them into the reconstruction module.

8. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 7 is characterized in that: The operation performed by the space-time shift module includes the steps of: T21, spatially shift a set of input aggregate features to obtain spatially shifted features; T22, apply the window multi-head self-attention mechanism to the spatial shift features and the current frame f i A part of each spatial shift feature generates multiple local windows, and generates a corresponding K value matrix and V value matrix for each local window of the spatial shift feature. i The local window generates the Q value matrix; T23, fuse the K value matrix, V value matrix and Q value matrix of each local window to obtain the local window fusion feature; All local window fusion features are fused to obtain the shifted aggregation features corresponding to the group of aggregation features.

9. The video defogging method based on a multi-scale spatiotemporal fusion network according to claim 8 is characterized in that: For the current frame f i The previous frame f i-1 The first group of aggregated features, step T21 specifically includes the steps of: T211, the current frame f i Divided into f i a and f i b Two parts, the previous frame f i-1 Divided into and Two parts; T212, group the features Slice along the channel dimension to obtain M feature slices, where the mth feature slice is expressed as Where m=1,....,M is the slice index; T213. For each feature slice Shift it spatially in the x and y directions to obtain the translated feature piece Group all features along the channel dimension Connect in series to get the spatial displacement characteristics T214, f i b and Carry out series connection, and compare the series connection results and spatial displacement characteristics After convolution, add the concatenated result, convolve the added result and add the added result again to obtain a first feature map; T215, yes After convolution and Add, convolve the added result and add it again to obtain the second feature map; T216, connecting the first feature map and the second feature map in series to obtain spatial shift features of a first group of aggregated features; T217, f i a As the current frame f i Multi-head self-attention mechanism for a part of the input window.

10. Video defogging system based on multi-scale spatiotemporal fusion network, characterized by: The invention is provided with an intelligent agent, which is used to execute the video defogging method based on a multi-scale spatiotemporal fusion network as described in any one of claims 1 to 9.

Images