Monitoring video real-time super-resolution reconstruction method and system fused with space-time attention mechanism

Abstract

The invention discloses a monitoring video real-time super-resolution reconstruction method and system fused with a space-time attention mechanism, and the method comprises the steps: firstly carrying out the extraction of shallow features, then carrying out the extraction of deep features, and carrying out the fusion of the shallow features and the deep features, so as to enhance the capturing capability of details in a monitoring video; the continuity and consistency of the video frames are enhanced by using time information between continuous frames, and the information of the time dimension is fused into the processing of the video frames, so that the dynamic capture capability and the overall video quality of a video monitoring system are improved; the time dimension is introduced into the SwinV3Transform, and the time information is integrated by modifying the attention mechanism, so that the time and space characteristics can be captured more accurately, and the continuity and visual effect of the video are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surveillance video processing, and in particular to a surveillance video real-time super-resolution reconstruction method and system integrating a spatiotemporal attention mechanism. Background Art

[0002] In recent years, with the rapid advancement of electronic technology and the continuous increase in social security needs, video surveillance technology has developed rapidly and has been widely used in many fields. Modern video surveillance systems are not limited to traditional security purposes, they are also integrated into urban management, traffic monitoring, commercial activities and educational facilities to improve operational efficiency and safety. The popularity of this technology is supported by high-performance computing hardware, which enables video surveillance to achieve real-time data processing and high-definition image capture, thus meeting the ever-changing technical standards and social needs.

[0003] The widespread application of video surveillance technology has significantly improved the safety and efficiency of society. It enables us to monitor key locations around the clock, reduce manpower requirements, save costs, and record and replay important events in real time. However, in order to ensure the monitoring effect, high-quality images and videos are required to meet the needs of security monitoring. There are currently some traditional super-resolution methods that can improve the video quality of video surveillance, such as bilinear interpolation and bicubic interpolation. These methods are simple to implement, have high computational efficiency, and are easy to deploy on systems that do not have high-performance processors; compared with deep learning models that require a lot of data training, traditional methods do not require complex training processes, so the demand for hardware resources is relatively low, which makes them more suitable for use in resource-constrained environments; due to the simpler calculation process, traditional super-resolution technology can process video streams faster and can support the needs of real-time video surveillance systems. But we still face the following problems:

[0004] 1) Feature extraction limitations: Traditional techniques usually rely on simpler feature extraction methods, such as basic convolutional layers, which limits their ability to handle complex scenes and capture details.

[0005] 2) Lack of effective time dimension processing. Many traditional methods fail to effectively integrate the time dimension of videos, resulting in insufficient performance in dynamic scenes and inability to effectively handle motion blur and time continuity issues in videos.

[0006] 3) Insufficient reconstruction quality. Traditional methods often fail to optimize both resolution improvement and visual quality when reconstructing high-definition videos, especially when processing scenes such as surveillance videos that are highly sensitive to details.

[0007] Therefore, there is an urgent need to study more effective super-resolution technology enhanced video surveillance methods, while meeting the quality of monitoring images and videos, breaking the limitations of feature extraction and ensuring real-time video super-resolution, especially in the application of video surveillance, significantly improving the performance and practicality of video super-resolution, showing incomparable advantages over traditional technologies. Summary of the invention

[0008] Based on the problems raised by the above background technology, the purpose of the present invention is to provide a real-time super-resolution reconstruction method and system for surveillance video integrating a spatiotemporal attention mechanism, which solves the problems of current real-time super-resolution reconstruction of surveillance video, such as limitation of feature extraction, lack of effective time dimension processing and insufficient reconstruction quality.

[0009] The present invention is achieved through the following technical solutions:

[0010] The first aspect of the present invention provides a surveillance video real-time super-resolution reconstruction method integrating a spatiotemporal attention mechanism, comprising the following steps:

[0011] Step S1, obtaining a surveillance video, and sequentially performing preprocessing and frame interpolation processing on the surveillance video to obtain a video frame;

[0012] Step S2: extract shallow features from the video frame using a deep separable convolutional network to obtain shallow features;

[0013] Step S3, determining a time window, and expanding the 2D feature blocks in the shallow features to 3D feature blocks based on the time window;

[0014] Step S4, using the SwinV3 Transformer network to extract deep features from the 3D feature blocks to obtain deep features;

[0015] Step S5: Fusing the shallow features and the deep features to obtain fused features, and using the fused features to reconstruct the video.

[0016] In the above technical solution, surveillance video is collected, and the collected surveillance video is preprocessed and frame interpolated. The purpose of preprocessing is to prepare clear and accurate video data for subsequent super-resolution processing, and the purpose of frame interpolation is to increase the frame rate of the surveillance video to improve the resolution and smoothness of the surveillance video.

[0017] At present, simple feature extraction methods are usually used to extract features from surveillance videos, such as basic convolutional layers, which limits their ability to process complex scenes and capture details. To this end, this method proposes to extract shallow features first and then deep features, and enhances the ability to capture details in surveillance videos by fusing shallow features with deep features. Among them, shallow feature extraction is to capture basic information such as color, texture, and edge in video frames. In this method, a deep separable convolutional network (DS-CNN) is used to extract shallow features from video frames. In the field of video surveillance, due to the real-time requirements, the use of DS-CNN can effectively reduce the computational burden and improve the processing speed, which is more suitable for the processing of real-time video streams. The lightweight design of DS-CNN is also suitable for deployment on edge devices, such as security surveillance cameras, which helps to perform fast video analysis near the data source.

[0018] At present, video data lacks effective temporal dimension processing. Many traditional methods fail to effectively integrate the temporal dimension of videos, resulting in insufficient performance in dynamic scenes and inability to effectively handle motion blur and temporal continuity problems in videos. Therefore, in this method, in order to enable deep feature extraction to handle temporal continuity in video sequences, temporal feature integration of shallow features is required after shallow feature extraction. Considering that the continuity and relevance of videos are very important in video surveillance, using the temporal information between consecutive frames to enhance the continuity and consistency of video frames and integrating the temporal dimension information into the processing of video frames is crucial to improving the dynamic capture capability and overall video quality of video surveillance systems.

[0019] The traditional sliding window Transformer (Swin Transformer) is mainly used to process the spatial information of images. SwinV2 Transformer is an upgrade of Swin Transformer, which can improve the processing power and resolution of images. The optimized self-attention mechanism can better perform feature extraction and recognition when processing high-resolution images. On this basis, this method proposes SwinV3 Transformer. The deep feature extraction in this method introduces the time dimension in SwinV3 Transformer and integrates the time information by modifying the attention mechanism, which can more accurately capture the time and space features and improve the coherence and visual effects of the video.

[0020] After deep features are extracted, shallow features and deep features are fused, and the fused feature map is used to reconstruct high-quality video output from low-resolution video frames and then complete the reconstruction of high-definition video. This process not only improves the visual quality of the video, but also enhances the recognizability of details, which is particularly important for fields such as security monitoring.

[0021] In an optional embodiment, the preprocessing and frame interpolation processing of the surveillance video in sequence includes the following steps:

[0022] Step S11, decoding the surveillance video into a continuous frame sequence, performing color space conversion on the continuous frame sequence, and performing noise reduction processing on the continuous frame sequence after the color space conversion using a noise reduction algorithm to obtain continuous frames;

[0023] Step S12: performing motion estimation on the continuous frames, generating intermediate frames between the continuous frames by using an interpolation algorithm according to the result of the motion estimation, and performing motion compensation on the intermediate frames. The continuous frames and the intermediate frames together constitute a video frame.

[0024] In an optional embodiment, using a deep separable convolutional network to extract shallow features from the video frame includes the following steps:

[0025] Step S21, using a sliding window technology to decompose the video frame into data blocks, and performing normalization processing on the data blocks;

[0026] Step S22, inputting the normalized data block into the deep separable convolutional network for shallow feature extraction; wherein the deep separable convolutional network includes two deep separable convolutional layers, a pooling layer and a fully connected layer;

[0027] The two-layer depth-separable convolutional layer performs feature processing on the normalized data block to obtain a feature map;

[0028] The pooling layer performs pooling processing on the feature map to obtain a 2D feature block;

[0029] The fully connected layer integrates the features of the 2D feature blocks to obtain shallow features.

[0030] In an optional embodiment, the two-layer depth-separable convolutional layer includes a first layer of convolutional kernels and a second layer of convolutional kernels;

[0031] Among them, the first layer of convolution kernel is a 3x3 convolution kernel, and the second layer of convolution kernel is a 5x5 convolution kernel;

[0032] The first convolution kernel performs feature extraction on the normalized data block to obtain a first feature, and activates the first feature using a ReLU activation function to obtain a first feature map;

[0033] The second layer convolution kernel extracts features from the first feature map to obtain a second feature, and activates the second feature using a ReLU activation function to obtain a second feature map.

[0034] In an optional embodiment, expanding the 2D feature block in the shallow feature to a 3D feature block based on the time window comprises the following steps:

[0035] Step S31: Obtain the t-th time frame in the video frame and obtain a 2D feature block F by shallow feature extraction t ;

[0036] Step S32: Acquire the 2D feature block F based on the time window t Adjacent 2D feature blocks of adjacent time frames;

[0037] Step S33: convert the 2D feature block F t The adjacent 2D feature blocks are stacked to generate a 3D feature block.

[0038] In an optional embodiment, using the SwinV3 Transformer network to perform deep feature extraction on the 3D feature block includes the following steps:

[0039] Step S41, using a window division technique to perform window division on the 3D feature block to obtain a pixel block;

[0040] Step S42, the SwinV3 Transformer network includes a plurality of SwinV3 Transformer blocks connected in sequence, a time matrix is ​​constructed, and the time matrix is ​​used in the SwinV3 Transformer block to perform deep feature extraction on the pixel block to obtain deep features;

[0041] Among them, each SwinV3 Transformer block is residually connected with the shallow features extracted by the deep separable convolutional network.

[0042] In an optional embodiment, constructing the time matrix includes the following steps:

[0043] Calculating the time difference between any two time frames in the video frame;

[0044] Convert the time difference into a weight value through a mapping function, and construct a first matrix with the weight value;

[0045] The first matrix is ​​normalized to obtain a time matrix.

[0046] In an optional embodiment, performing deep feature extraction on the pixel block using the time matrix in the SwinV3 Transformer block comprises the following steps:

[0047] The SwinV3 Transformer block includes several SwinV3 Transformer layers, in which a spatiotemporal joint multi-head self-attention mechanism is introduced, and the spatiotemporal joint multi-head self-attention mechanism is used to calculate the spatial dimension and the temporal dimension of the pixel block to obtain spatiotemporal features; wherein the spatiotemporal joint multi-head self-attention mechanism is implemented by introducing a time matrix to modify the self-attention calculation formula;

[0048] The spatiotemporal features are refined through layer normalization and multi-layer perceptron to obtain deep features.

[0049] In an optional embodiment, the shallow features and the deep features are fused to obtain fused features, and video reconstruction using the fused features includes the following steps:

[0050] Step S51, performing weighted calculation on the shallow features and the deep features to obtain weighted fusion features;

[0051] Step S52: up-sample the weighted fusion features using transposed convolution to obtain an up-sampled video;

[0052] Step S53, calculating the residual mapping using the residual learning mechanism;

[0053] Step S54: Fusing the residual map with the upsampled video to obtain a high-resolution video.

[0054] The second aspect of the present invention provides a surveillance video real-time super-resolution reconstruction system integrating a spatiotemporal attention mechanism, comprising:

[0055] An acquisition module, the acquisition module is used to acquire surveillance video, and perform preprocessing and frame interpolation processing on the surveillance video in sequence to obtain video frames;

[0056] A shallow feature extraction module, wherein the shallow feature extraction module is used to extract shallow features of the video frame using a deep separable convolutional network to obtain shallow features;

[0057] A time feature integration module, the time feature integration module is used to determine a time window, and expand the 2D feature block in the shallow feature to a 3D feature block based on the time window;

[0058] A deep feature extraction module, wherein the deep feature extraction module is used to perform deep feature extraction on the 3D feature block using SwinV3 Transformer to obtain deep features;

[0059] The video reconstruction module is used to fuse the shallow features and the deep features to obtain fused features, and use the fused features to reconstruct the video.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. Extract shallow features first, then deep features. By fusing shallow features with deep features, the ability to capture details in surveillance videos is enhanced.

[0062] 2. Use the time information between consecutive frames to enhance the continuity and consistency of video frames, integrate the time dimension information into the processing of video frames, and improve the dynamic capture capability and overall video quality of the video surveillance system;

[0063] 3. By introducing the time dimension in SwinV3 Transformer and integrating time information by modifying the attention mechanism, we can more accurately capture the temporal and spatial features and improve the coherence and visual effects of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0065] Figure 1 A schematic diagram of a flow chart of a method for real-time super-resolution reconstruction of surveillance video integrating a spatiotemporal attention mechanism provided in Example 1 of the present invention;

[0066] Figure 2 A schematic diagram of the overall structure of the video surveillance super-resolution provided in Example 1 of the present invention;

[0067] Figure 3 A schematic diagram of the structure of the residual SwinV3 Transformer block provided in Example 1 of the present invention;

[0068] Figure 4 A schematic diagram of the structure of the SwinV3 Transformer layer provided in Example 1 of the present invention;

[0069] Figure 5A schematic diagram of the structure of the spatiotemporal joint multi-head self-attention mechanism provided in Example 1 of the present invention;

[0070] Figure 6 A schematic diagram of the structure of a real-time super-resolution reconstruction system for surveillance video that integrates a spatiotemporal attention mechanism provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0072] Example 1

[0073] Figure 1 The flowchart of the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism provided in Example 1 of the present invention is as follows: Figure 1 As shown in FIG, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism includes the following steps:

[0074] Step S1, obtaining a surveillance video, and sequentially performing preprocessing and frame interpolation processing on the surveillance video to obtain a video frame;

[0075] Step S2: extract shallow features from the video frame using a deep separable convolutional network to obtain shallow features;

[0076] Step S3, determining a time window, and expanding the 2D feature blocks in the shallow features to 3D feature blocks based on the time window;

[0077] Step S4, using the SwinV3 Transformer network to extract deep features from the 3D feature blocks to obtain deep features;

[0078] Step S5: Fusing the shallow features and the deep features to obtain fused features, and using the fused features to reconstruct the video.

[0079] It should be noted that the framework of this method is Figure 2 As shown, surveillance video is collected, and preprocessed and frame interpolated. The purpose of preprocessing is to prepare clear and accurate video data for subsequent super-resolution processing. The purpose of frame interpolation is to increase the frame rate of the surveillance video to improve the resolution and smoothness of the surveillance video.

[0080] At present, simple feature extraction methods are usually used to extract features from surveillance videos, such as basic convolutional layers, which limits their ability to process complex scenes and capture details. To this end, this method proposes to extract shallow features first and then deep features, and enhances the ability to capture details in surveillance videos by fusing shallow features with deep features. Among them, shallow feature extraction is to capture basic information such as color, texture, and edge in video frames. In this method, a deep separable convolutional network (DS-CNN) is used to extract shallow features from video frames. In the field of video surveillance, due to the real-time requirements, the use of DS-CNN can effectively reduce the computational burden and improve the processing speed, which is more suitable for the processing of real-time video streams. The lightweight design of DS-CNN is also suitable for deployment on edge devices, such as security surveillance cameras, which helps to perform fast video analysis near the data source.

[0081] At present, video data lacks effective temporal dimension processing. Many traditional methods fail to effectively integrate the temporal dimension of videos, resulting in insufficient performance in dynamic scenes and inability to effectively handle motion blur and temporal continuity problems in videos. Therefore, in this method, in order to enable deep feature extraction to handle temporal continuity in video sequences, temporal feature integration of shallow features is required after shallow feature extraction. Considering that the continuity and relevance of videos are very important in video surveillance, using the temporal information between consecutive frames to enhance the continuity and consistency of video frames and integrating the temporal dimension information into the processing of video frames is crucial to improving the dynamic capture capability and overall video quality of video surveillance systems.

[0082] The traditional sliding window Transformer (Swin Transformer) is mainly used to process the spatial information of images. SwinV2 Transformer is an upgrade of Swin Transformer, which can improve the processing power and resolution of images. The optimized self-attention mechanism can better perform feature extraction and recognition when processing high-resolution images. On this basis, this method proposes SwinV3 Transformer. The deep feature extraction in this method introduces the time dimension in SwinV3 Transformer and integrates the time information by modifying the attention mechanism, which can more accurately capture the time and space features and improve the coherence and visual effects of the video.

[0083] After deep features are extracted, shallow features and deep features are fused, and the fused feature map is used to reconstruct high-quality video output from low-resolution video frames and then complete the reconstruction of high-definition video. This process not only improves the visual quality of the video, but also enhances the recognizability of details, which is particularly important for fields such as security monitoring.

[0084] In an optional embodiment, the preprocessing and frame interpolation processing of the surveillance video in sequence includes the following steps:

[0085] Step S11, decoding the surveillance video into a continuous frame sequence, performing color space conversion on the continuous frame sequence, and performing noise reduction processing on the continuous frame sequence after the color space conversion using a noise reduction algorithm to obtain continuous frames;

[0086] Step S12: performing motion estimation on the continuous frames, generating intermediate frames between the continuous frames by using an interpolation algorithm according to the result of the motion estimation, and performing motion compensation on the intermediate frames. The continuous frames and the intermediate frames together constitute a video frame.

[0087] It should be noted that in the embodiments of the present invention, monitoring videos are obtained through direct camera acquisition, network video acquisition, mobile device integration, and cloud infrastructure.

[0088] The surveillance video is preprocessed and frame interpolated in sequence, where the preprocessing includes decoding, color conversion and noise reduction. First, the surveillance video file is decoded into a continuous frame sequence so that the original data can be effectively read and used by subsequent algorithms; then the continuous frame sequence is converted into a color space from RGB color space to YUV color space, because YUV color space can be more efficient in processing brightness and chrominance; finally, the continuous frame sequence is subjected to noise reduction to reduce noise and artifacts in the compressed video, providing a clearer basic image for super-resolution processing.

[0089] Frame interpolation processing includes motion estimation, generation of intermediate frames, and motion compensation. First, use the optical flow algorithm or block-based matching algorithm to estimate the motion between consecutive frames and determine the movement path of the pixels. Motion estimation is the core of frame interpolation, which can predict and compensate for image changes caused by object motion or camera movement; then, based on the results of motion estimation, intermediate frames are generated through interpolation methods, which can significantly increase the frame rate of the video, improve the smoothness and temporal resolution of the video, and are especially suitable for the production of slow motion effects or the improvement of video quality; finally, motion compensation is performed on the predicted intermediate frames, and the pixel positions within the frames are adjusted to reduce the blur and distortion caused by motion, which can ensure that the newly synthesized video frames are visually consistent with the original frames without obvious jumps or blurring.

[0090] In an optional embodiment, using a deep separable convolutional network to extract shallow features from the video frame includes the following steps:

[0091] Step S21, using a sliding window technology to decompose the video frame into data blocks, and performing normalization processing on the data blocks;

[0092] Step S22, inputting the normalized data block into the deep separable convolutional network for shallow feature extraction; wherein the deep separable convolutional network includes two deep separable convolutional layers, a pooling layer and a fully connected layer;

[0093] The two-layer depth-separable convolutional layer performs feature processing on the normalized data block to obtain a feature map;

[0094] The pooling layer performs pooling processing on the feature map to obtain a 2D feature block;

[0095] The fully connected layer integrates the features of the 2D feature blocks to obtain shallow features.

[0096] It should be noted that when performing shallow feature extraction, the video frame must first be decomposed into smaller data blocks so that local features can be effectively captured during feature extraction. Specifically, the sliding window method is applied in each frame to decompose the video frame into 32x32 pixel data blocks. This size of data block can balance the computational complexity and the ability to capture sufficient details, and is suitable for a variety of video surveillance scenarios, thereby effectively capturing local features.

[0097] Furthermore, in practical applications, there may be overlapping parts between each data block, and the step size is usually smaller than the window size, which can increase the spatial continuity between the data blocks and reduce the edge effect. In order to ensure the coverage and overlap of the feature map in the embodiment of the present invention, the step size is set to 16 pixels, which can ensure that there is a 50% overlap between each data block, which helps to reduce the edge effect and improve the feature coherence.

[0098] Furthermore, each independent data block is normalized by using Z-score normalization to subtract the average value of the block from the pixel value of each data block and divide it by its standard deviation. This normalization can further help the network learn more stably, especially in surveillance videos with large changes in lighting and background of video frames.

[0099] Specifically, the calculation formula for Z-score standardization is as follows:

[0100]

[0101] Where X is the pixel value of an independent data block, μ is the average value of the data block, and σ is the standard deviation of the data block.

[0102] Constructing a deep separable convolutional network, the DS-CNN convolutional network can significantly reduce the number of network parameters and computational costs while maintaining effective feature extraction capabilities. Among them, the deep separable convolutional network includes two layers of deep separable convolutional layers, a pooling layer, and a fully connected layer. The two layers of deep separable convolutional layers are sufficient to handle preliminary feature extraction tasks. After the features are activated, feature maps (Feature Maps) are formed. These feature maps represent the feature representations of the original input at different spatial scales. In the present invention, the maximum pooling operation is applied through the pooling layer, a block with a pooling size of 2x2 is selected, and the maximum value of the coverage area is selected as the representative feature of the area, reducing the feature dimension while retaining important feature information, in order to reduce the computational complexity and improve the abstraction ability of the feature. Finally, the extracted 2D feature blocks are integrated through the fully connected layer and prepared to be input into the deep feature extraction network. This step ensures that the information obtained from the shallow network can be effectively transferred to the deep network, providing a basis for subsequent complex feature extraction and video reconstruction.

[0103] In an optional embodiment, the two-layer depth-separable convolutional layer includes a first layer of convolutional kernels and a second layer of convolutional kernels;

[0104] Among them, the first layer of convolution kernel is a 3x3 convolution kernel, and the second layer of convolution kernel is a 5x5 convolution kernel;

[0105] The first convolution kernel performs feature extraction on the normalized data block to obtain a first feature, and activates the first feature using a ReLU activation function to obtain a first feature map;

[0106] The second layer convolution kernel extracts features from the first feature map to obtain a second feature, and activates the second feature using a ReLU activation function to obtain a second feature map.

[0107] It should be noted that the convolution kernel size is 3x3 in the first layer and 5x5 in the second layer. The ReLU activation function is used after each convolution to improve the nonlinear processing capability, and a batch normalization layer is added after each convolution operation to speed up the training process and improve the generalization ability of the model.

[0108] In an optional embodiment, expanding the 2D feature block in the shallow feature to a 3D feature block based on the time window comprises the following steps:

[0109] Step S31: Obtain the t-th time frame in the video frame and obtain a 2D feature block F by shallow feature extraction t ;

[0110] Step S32: Acquire the 2D feature block F based on the time window t Adjacent 2D feature blocks of adjacent time frames;

[0111] Step S33: convert the 2D feature block F t The adjacent 2D feature blocks are stacked to generate a 3D feature block.

[0112] It should be noted that the 2D feature blocks obtained from the shallow feature extraction stage are extended to 3D feature blocks in order to capture the temporal continuity and dynamic changes in the video sequence. First, feature blocks on adjacent time frames are obtained from continuous video frames, and the 2D feature block F is obtained by shallow feature extraction at the tth time frame in the video frame. t For example, the size of each feature block is Hx W x C, where H, W and C represent the height, width and number of channels of the block respectively. Here, the sizes of H and W are both 32 pixels, and the number of channels is 3 YUV channels.

[0113] Secondly, a time window is selected, which defines the number of consecutive frames used to generate 3D blocks. The selection of the time window is critical when performing 3D block segmentation. The time window T′T defines the number of consecutive frames used to form 3D blocks, which is crucial for capturing dynamic changes and temporal continuity in video sequences. The frame rate of the surveillance video will affect the selection of T′T. High frame rate videos allow for smaller T′T values ​​because the changes between adjacent frames are small. Low frame rate videos may require larger T′T values ​​to capture sufficient motion information. Scene dynamics in surveillance videos, such as fast-moving objects, may require larger T′T to ensure that sufficient motion information is included. Larger T′T values ​​may increase processing requirements and latency, but can improve the accuracy of dynamic capture. It is necessary to balance the real-time processing requirements with performance. In most surveillance applications, considering real-time and efficiency, the recommended T′T value range is 3 to 5. This range is usually sufficient to capture key dynamics while maintaining real-time processing. For high-speed moving object monitoring, such as traffic monitoring, T′T=5 or higher may be considered to ensure that important motion information is not missed; for relatively static or slowly changing scenes, such as indoor monitoring, T′T=3 is sufficient.

[0114] In this embodiment, T′T=3 is taken as an example. When T′T=3, it means that each 3D feature block will cover the current frame and one frame before and after it. For the 2D feature block F t , the adjacent 2D feature blocks of adjacent time frames obtained are F t-1 and F t+1 , by stacking F t-1 , F t and F t+1Construct the 3D feature block B of the tth time frame t , so that the dimension of the 3D feature block will become H x W x (C x T′T), and the dimension in this embodiment is 32x32x(3x3).

[0115] Furthermore, the formula for constructing 3D feature blocks is:

[0116] B t =[F t-1 ; F t ; F t+1 ]

[0117] Among them, F t-1 , F t+1 They are the feature blocks of the previous and next time frames (t-1 and t+1 time frames), and the semicolon indicates the stacking process.

[0118] Furthermore, for the start and end frames of the video sequence, since there is no previous frame or next frame, the strategy of filling the same frame can be used to construct the 3D block. For the first frame:

[0119] B1=[F1;F1;F2]

[0120] For the last frame:

[0121] B end =[F end-1 ; F end ; F end ]

[0122] In an optional embodiment, using the SwinV3 Transformer network to perform deep feature extraction on the 3D feature block includes the following steps:

[0123] Step S41, using a window division technique to perform window division on the 3D feature block to obtain a pixel block;

[0124] Step S42, the SwinV3 Transformer network includes a plurality of SwinV3 Transformer blocks connected in sequence, a time matrix is ​​constructed, and the time matrix is ​​used in the SwinV3 Transformer block to perform deep feature extraction on the pixel block to obtain deep features;

[0125] Among them, each SwinV3 Transformer block is residually connected with the shallow features extracted by the deep separable convolutional network.

[0126] It should be noted that the input 3D feature block is subjected to standard window segmentation, wherein each window contains an MxM pixel block, and in this embodiment, 8x8 or 16x16 is taken. The processing of dividing the 3D feature block into small windows usually requires less computing resources, which is particularly important for real-time video surveillance systems. By further dividing the 3D feature block, real-time image analysis can be performed while maintaining a low computing cost, and the accuracy and efficiency of the video surveillance system can be improved, especially in application scenarios that require high-quality image analysis. This method can significantly improve the monitoring effect and practicality.

[0127] SwinV3 Transformer network Figure 3 As shown, the SwinV3 Transformer network includes several SwinV3 Transformer blocks connected in sequence, feature extraction is performed in each SwinV3 Transformer block, and several SwinV3 Transformer blocks connected in sequence perform repeated feature extraction in sequence.

[0128] Each SwinV3 Transformer block is residually connected to the shallow features extracted by the deep separable convolutional network. At this time, the SwinV3 Transformer block becomes a residual SwinV3 Transformer block (RSV3TB). In the deep network, the residual connection helps maintain the healthy flow of gradients, thereby avoiding the problem of gradient disappearance during back propagation. By repeatedly extracting features, the complexity and usefulness of features can be gradually improved.

[0129] Furthermore, the update formula of RSV3TB output is:

[0130] O t =LayerNorm(STMH-SA(LN(B t ))+B t )

[0131] O t =LayerNorm(MLP(LN(O t ))+O t )

[0132] Among them, t is the output feature, B t denotes input block, LN denotes layer normalization, and MLP denotes multilayer perceptron (a feedforward neural network).

[0133] In an optional embodiment, constructing the time matrix includes the following steps:

[0134] Calculating the time difference between any two time frames in the video frame;

[0135] Convert the time difference into a weight value through a mapping function, and construct a first matrix with the weight value;

[0136] The first matrix is ​​normalized to obtain a time matrix.

[0137] It should be noted that in order to incorporate the time dimension into the self-attention mechanism, it is proposed to introduce the time matrix T. First, for each frame in the video sequence, the time difference between it and other frames is calculated. These time differences can be simply expressed as the difference between frame indices. For example, the time difference between time frame i and time frame j is |i-j|; secondly, the time difference is converted into a weight in the time matrix through a mapping function, such as using an exponential decay function to enhance the influence of neighboring frames and reduce the influence of distant frames:

[0138] T i,j =e -α|i-j|

[0139] Among them, α is the decay rate, which controls the influence of time distance.

[0140] In order to make the time matrix effective and stable under different video sequence lengths, it is necessary to normalize it so that the sum of each column is 1, so that the sum of the attention distribution can be kept to 1 when applying the time matrix; finally, in the self-attention calculation, the traditional QK T The term represents the similarity between the query and the key (where K T represents the transpose of matrix K), and now the time matrix T is added as a regulation item. In this way, the time matrix can adjust the influence between frames according to the time interval between them, so that the attention mechanism not only considers the similarity of spatial features, but also the influence of temporal continuity. By introducing the time matrix T, T becomes a bridge connecting temporal information and spatial feature processing, which enhances the model's understanding of video data.

[0141] In an optional embodiment, performing deep feature extraction on the pixel block using the time matrix in the SwinV3 Transformer block comprises the following steps:

[0142] The SwinV3 Transformer block includes several SwinV3 Transformer layers, in which a spatiotemporal joint multi-head self-attention mechanism is introduced, and the spatiotemporal joint multi-head self-attention mechanism is used to calculate the spatial dimension and the temporal dimension of the pixel block to obtain spatiotemporal features; wherein the spatiotemporal joint multi-head self-attention mechanism is implemented by introducing a time matrix to modify the self-attention calculation formula;

[0143] The spatiotemporal features are refined through layer normalization and multi-layer perceptron to obtain deep features.

[0144] It should be noted that the SwinV3 Transformer layer Figure 4 As shown, it includes a spatiotemporal joint multi-head self-attention mechanism, layer normalization, and a multi-layer perceptron (MLP).

[0145] Among them, the structure of the spatio-temporal multi-head self-attention mechanism (STMH-SA) is as follows Figure 5 As shown in the figure, this mechanism can simultaneously process the spatial dimension (the relationship between pixels in an image) and the temporal dimension (the relationship between frames), which is achieved by modifying the calculation formula of self-attention and introducing the weight calculation of the temporal dimension. SwinV3 Transformer applies STMH-SA independently on each window, and in consecutive layers, the position of the window will be offset (usually half the window size) to allow information exchange across windows and enhance the network's ability to capture long-distance dependencies.

[0146] Further, spatiotemporal joint multi-head self-attention calculation:

[0147]

[0148] Where Q, K, and V represent the query vector, key vector, and value vector. T is a time-dependent matrix used to adjust the influence between different frames. k is the dimension of the key vector.

[0149] Furthermore, in successive layers, the position of the window is offset (usually by half the window size) to allow cross-window information exchange and enhance the network’s ability to capture long-distance dependencies. Assume that k is the sequence number of the SwinV3 Transformer layer, W k represents the window used in the kth layer, then the window offset can be expressed as:

[0150] W t+1 =shift(Wt ,offset)

[0151] Where shift() represents the window offset operation, and offset is the window offset, which is usually half of the window size.

[0152] Therefore, the input in each layer should be the offset window, and the integrated features are:

[0153] STMH-SA(Q,K,V)=Attention(Q,K,V)

[0154] (Q,K,V)=shift(W,offset)

[0155] Where (Q, K, V) is obtained through the offset window.

[0156] In the SwinV3 Transformer layer, the features processed by SW-MSA will undergo a series of layer normalization (LN) and multilayer perceptron (MLP) operations to further extract and refine the features.

[0157] In an optional embodiment, the shallow features and the deep features are fused to obtain fused features, and video reconstruction using the fused features includes the following steps:

[0158] Step S51, performing weighted calculation on the shallow features and the deep features to obtain weighted fusion features;

[0159] Step S52: up-sample the weighted fusion features using transposed convolution to obtain an up-sampled video;

[0160] Step S53, calculating the residual mapping using the residual learning mechanism;

[0161] Step S54: Fusing the residual map with the upsampled video to obtain a high-resolution video.

[0162] It should be noted that after deep features are extracted, shallow features and deep features are fused, and the weighted sum method is used to integrate features at different levels, where F combined is the fused feature:

[0163] F combined =α×F shallow +β×F deep

[0164] Among them, α and β are the weight parameters of the learned shallow features and deep features, which are used to balance the shallow features Fshallow and deep features F deep contribution.

[0165] Use transposed convolution to transform the fused feature F combined Upsampling is performed to restore high-resolution vision. The upsampling operation can be expressed as:

[0166] F up =Conv T (F combined ,W up , stride=s)

[0167] Among them, Conv T represents the transposed convolution operation, W up is the weight of the transposed convolution, and tride=s defines the upsampling ratio.

[0168] Furthermore, in order to restore video details more finely and reduce reconstruction errors, a residual learning mechanism is introduced. The residual mapping can be expressed as:

[0169] R=F HR -Upsample(F LR )

[0170] Among them, F HR is the reconstructed high-resolution video, which is obtained by reconstructing the high-resolution output of S54. LR It is a low-resolution input, which is obtained by preprocessing the captured video. Upsample is a simple upsampling operation.

[0171] Add the residual map back to the upsampled video to complete the reconstruction of the high-resolution video:

[0172] I HR =F up +R

[0173] Among them, I HR The final high-resolution video output is obtained through the above steps. Through the above steps, high-quality video output can be effectively reconstructed from low-resolution video frames, which is suitable for application scenarios that require high-resolution video surveillance. This process not only improves the visual quality of the image, but also enhances the recognizability of details, which is particularly important for security monitoring and other fields.

[0174] Example 2

[0175] Figure 6 The schematic diagram of the structure of the real-time super-resolution reconstruction system of surveillance video integrating the spatiotemporal attention mechanism provided in Example 2 of the present invention is as follows: Figure 6As shown in the figure, the real-time super-resolution reconstruction system of surveillance video integrating the spatiotemporal attention mechanism includes:

[0176] An acquisition module, the acquisition module is used to acquire surveillance video, and perform preprocessing and frame interpolation processing on the surveillance video in sequence to obtain video frames;

[0177] A shallow feature extraction module, wherein the shallow feature extraction module is used to extract shallow features of the video frame using a deep separable convolutional network to obtain shallow features;

[0178] A time feature integration module, the time feature integration module is used to determine a time window, and expand the 2D feature block in the shallow feature to a 3D feature block based on the time window;

[0179] A deep feature extraction module, wherein the deep feature extraction module is used to perform deep feature extraction on the 3D feature block using SwinV3 Transformer to obtain deep features;

[0180] The video reconstruction module is used to fuse the shallow features and the deep features to obtain fused features, and use the fused features to reconstruct the video.

[0181] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Real-time super-resolution reconstruction method of surveillance video integrating spatiotemporal attention mechanism, It is characterized in that The steps include: Step S1, obtaining a surveillance video, and sequentially performing preprocessing and frame interpolation processing on the surveillance video to obtain a video frame; Step S2: extract shallow features from the video frame using a deep separable convolutional network to obtain shallow features; Step S3, determining a time window, and expanding the 2D feature blocks in the shallow features to 3D feature blocks based on the time window; Step S4, using the SwinV3 Transformer network to extract deep features from the 3D feature blocks to obtain deep features; Step S5: Fusing the shallow features and the deep features to obtain fused features, and using the fused features to reconstruct the video.

2. According to claim 1, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism, It is characterized in that The monitoring video is preprocessed and frame interpolated in sequence, comprising the following steps: Step S11, decoding the surveillance video into a continuous frame sequence, performing color space conversion on the continuous frame sequence, and performing noise reduction processing on the continuous frame sequence after the color space conversion using a noise reduction algorithm to obtain continuous frames; Step S12: performing motion estimation on the continuous frames, generating intermediate frames between the continuous frames by using an interpolation algorithm according to the result of the motion estimation, and performing motion compensation on the intermediate frames. The continuous frames and the intermediate frames together constitute a video frame.

3. According to claim 1, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism, It is characterized in that Using a deep separable convolutional network to extract shallow features from the video frame includes the following steps: Step S21, using a sliding window technology to decompose the video frame into data blocks, and performing normalization processing on the data blocks; Step S22, inputting the normalized data block into the deep separable convolutional network for shallow feature extraction; wherein the deep separable convolutional network includes two deep separable convolutional layers, a pooling layer and a fully connected layer; The two-layer depth-separable convolutional layer performs feature processing on the normalized data block to obtain a feature map; The pooling layer performs pooling processing on the feature map to obtain a 2D feature block; The fully connected layer integrates the features of the 2D feature blocks to obtain shallow features.

4. According to claim 3, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism, It is characterized in that The two-layer depth-separable convolutional layer includes a first layer of convolutional kernels and a second layer of convolutional kernels; Among them, the first layer of convolution kernel is a 3x3 convolution kernel, and the second layer of convolution kernel is a 5x5 convolution kernel; The first convolution kernel performs feature extraction on the normalized data block to obtain a first feature, and activates the first feature using a ReLU activation function to obtain a first feature map; The second layer convolution kernel extracts features from the first feature map to obtain a second feature, and activates the second feature using a ReLU activation function to obtain a second feature map.

5. According to claim 1, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism, It is characterized in that Expanding the 2D feature block in the shallow feature to a 3D feature block based on the time window comprises the following steps: Step S31: Obtain the t-th time frame in the video frame and obtain a 2D feature block F by shallow feature extraction t ; Step S32: Acquire the 2D feature block F based on the time window t Adjacent 2D feature blocks of adjacent time frames; Step S33: convert the 2D feature block F t The adjacent 2D feature blocks are stacked to generate a 3D feature block.

6. According to claim 1, the real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism, It is characterized in that Using the SwinV3 Transformer network to perform deep feature extraction on the 3D feature block includes the following steps: Step S41, using a window division technique to perform window division on the 3D feature block to obtain a pixel block; Step S42, the SwinV3 Transformer network includes a plurality of SwinV3 Transformer blocks connected in sequence, a time matrix is ​​constructed, and deep features are extracted from the pixel block using the time matrix in the SwinV3 Transformer block to obtain deep features; Among them, each SwinV3 Transformer block is residually connected with the shallow features extracted by the deep separable convolutional network.

7. The real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism according to claim 6, It is characterized in that Constructing a time matrix includes the following steps: Calculating the time difference between any two time frames in the video frame; Convert the time difference into a weight value through a mapping function, and construct a first matrix with the weight value; The first matrix is ​​normalized to obtain a time matrix.

8. The real-time super-resolution reconstruction method of surveillance video integrating the spatiotemporal attention mechanism according to claim 6, It is characterized in that Using the time matrix to perform deep feature extraction on the pixel block in the SwinV3 Transformer block comprises the following steps: The SwinV3 Transformer block includes several SwinV3 Transformer layers, in which a spatiotemporal joint multi-head self-attention mechanism is introduced, and the spatiotemporal joint multi-head self-attention mechanism is used to calculate the spatial dimension and the temporal dimension of the pixel block to obtain spatiotemporal features; wherein the spatiotemporal joint multi-head self-attention mechanism is implemented by introducing a time matrix to modify the self-attention calculation formula; The spatiotemporal features are refined through layer normalization and multi-layer perceptron to obtain deep features.

9. The real-time super-resolution reconstruction method of surveillance video integrating spatiotemporal attention mechanism according to claim 1, It is characterized in that The shallow features and the deep features are fused to obtain fused features, and video reconstruction using the fused features includes the following steps: Step S51, performing weighted calculation on the shallow features and the deep features to obtain weighted fusion features; Step S52: up-sample the weighted fusion features using transposed convolution to obtain an up-sampled video; Step S53, calculating the residual mapping using the residual learning mechanism; Step S54: Fusing the residual map with the upsampled video to obtain a high-resolution video.

10. Real-time super-resolution reconstruction system of surveillance video integrating spatiotemporal attention mechanism, It is characterized in that include: An acquisition module, the acquisition module is used to acquire surveillance video, and perform preprocessing and frame interpolation processing on the surveillance video in sequence to obtain video frames; A shallow feature extraction module, wherein the shallow feature extraction module is used to extract shallow features of the video frame using a deep separable convolutional network to obtain shallow features; A time feature integration module, the time feature integration module is used to determine a time window, and expand the 2D feature block in the shallow feature to a 3D feature block based on the time window; A deep feature extraction module, wherein the deep feature extraction module is used to perform deep feature extraction on the 3D feature block using SwinV3 Transformer to obtain deep features; The video reconstruction module is used to fuse the shallow features and the deep features to obtain fused features, and use the fused features to reconstruct the video.

Images