Video super-resolution method based on spatio-temporal convolution attention

Abstract

The application discloses a video super-resolution method based on space-time convolution attention, and is specifically implemented according to the following steps: step 1, converting an input high-resolution video sequence into a low-resolution video sequence, so as to obtain a low-resolution video set; step 2, constructing a deep neural network structure for video super-resolution; step 3, training the network based on paired video data; step 4, video super-resolution reconstruction, so as to realize super-resolution recovery of the input video sequence. The application solves the problem in the prior art that the network is regarded as a black box, performance is improved by increasing the network depth and parameter scale, the rapid growth of the calculation complexity is ignored, and the efficiency and the reconstruction quality are difficult to be considered.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a video super-resolution method based on spatiotemporal convolutional attention. Background Technology

[0002] In the mobile internet era, high-definition video has become the mainstream carrier of information transmission, playing a core role in key areas such as public security, satellite remote sensing, and digital entertainment. However, limited by factors such as the hardware performance, manufacturing cost, and transmission bandwidth of the shooting equipment, the video quality ultimately received by users often suffers from insufficient resolution, blurred details, and missing textures, severely restricting the performance of subsequent high-precision visual applications such as facial recognition, license plate analysis, and remote sensing interpretation. Therefore, developing video super-resolution technology that can restore details and maintain temporal continuity has significant practical significance and value.

[0003] Super-resolution is one of the fundamental ill-conditioned inverse problems in the field of image and video processing. Existing methods can be mainly divided into model-based methods and deep learning-based methods. Early methods were mainly model-based. These methods typically construct an objective function within a Bayesian inference framework, including data fidelity terms and image prior terms (such as gradient priors, self-similarity, manifold constraints, etc.), and obtain a high-resolution image through iterative optimization. Compared with single-image super-resolution, the key to video super-resolution lies in effectively fusing complementary information from multiple video frames. Therefore, model-based video super-resolution methods are usually based on single-frame reconstruction techniques, introducing motion estimation and motion compensation mechanisms to align and fuse information from adjacent frames. However, because the optimization process requires multiple iterations, and manually designed priors are difficult to fully characterize the structural characteristics of natural images, these methods generally suffer from low computational efficiency and limited prior modeling capabilities.

[0004] With the rapid advancement of computing hardware, deep learning-based video super-resolution methods have achieved significant performance improvements through data-driven approaches by learning end-to-end mappings from low-resolution sequences to high-resolution video frames, and have become the mainstream method. The core of neural network-based video super-resolution methods lies in designing network architectures that can fully exploit spatiotemporal correlations. These methods mainly fall into two categories: multi-frame fusion-based video super-resolution methods and recurrent neural network (RNN)-based methods. Multi-frame fusion-based methods further include explicit alignment-based methods, implicit alignment-based methods, and spatiotemporal joint modeling methods. Explicit alignment-based methods typically include modules such as motion estimation, inter-frame alignment, and feature fusion, aggregating temporal correlations through explicit alignment mechanisms. Implicit alignment-based methods utilize deformable convolutions or adaptive kernels to dynamically predict a set of offset sampling points or a convolutional kernel for each location, thereby implicitly fusing relevant information from adjacent frames. Spatiotemporal joint modeling-based methods treat the video as a spatiotemporal volume, using 3D convolutions or spatiotemporal attention mechanisms to aggregate spatiotemporal information. Furthermore, RNN-based methods utilize their memory units to recursively pass states along the time dimension, implicitly capturing long-term temporal dependencies and progressively reconstructing video frames. However, most existing studies treat the network as a black box, relying on increasing network depth and parameter size to improve performance, ignoring the rapid increase in computational complexity, creating a bottleneck where efficiency and reconstruction quality are difficult to balance. Summary of the Invention

[0005] The purpose of this invention is to provide a video super-resolution method based on spatiotemporal convolutional attention, which solves the problem in the prior art that treats the network as a black box, relies on increasing network depth and parameter scale to improve performance, ignores the rapid increase in computational complexity, and makes it difficult to balance efficiency and reconstruction quality.

[0006] The technical solution adopted in this invention is a video super-resolution method based on spatiotemporal convolutional attention, which is implemented according to the following steps:

[0007] Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; Step 2: Construct a deep neural network structure for video super-resolution; Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0008] The invention is further characterized in that, In step 1, for the given NA set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0009] In step 1, bicubic interpolation downsampling is performed on the high-resolution video sequence to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0010] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0011] The specific deep neural network structure in step 2 is as follows: First, the input low-resolution video sequence... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

[0012] Step 2 is implemented in the following steps: Step 2.1, Multi-frame feature extraction module: First, the input low-resolution video sequence... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced so that the network only needs to learn high-frequency residual details. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is output.

[0013] Each enhancement block in step 2.2 consists of two substructures cascaded in series. The feature map is first fed into the layer normalization and convolutional feedforward network, and then processed by the first substructure, i.e., the spatiotemporal window attention module, and the second substructure, i.e., the spatiotemporal convolutional attention module, in sequence. Finally, the feature map is transformed by the layer normalization and convolutional layer, and residual connection is made with the input to reconstruct the feature map. Spatiotemporal window attention module: This module receives the feature map processed by the normalization layer and convolutional feedforward network in step 2.2 as input. This input feature map contains the spatial local features and temporal series features of multiple frames of video. The module reconstructs these features into a unified vector space through spatiotemporal dimension projection technology, and uses the self-attention mechanism to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames. Spatiotemporal Convolutional Attention Module: This module receives feature maps from the preceding substructures in step 2.2 (such as the spatiotemporal window attention module) as input. First, it achieves information interaction between adjacent frames without increasing parameters through temporal shifting operations. Then, it splits the feature maps into a main branch and an auxiliary branch along the channel dimension. The main branch uses two parallel paths for feature extraction: one path uses the main branch features to generate a dynamic convolution kernel, and then implicitly aligns motion features and reduces dimensionality through depthwise convolution; the other path extracts static background information through shared large kernel convolution (kernel parameters are initialized by the network and dynamically updated), and fuses the temporal context through temporal aggregation operations. The two outputs are added together and concatenated with the auxiliary branch features to achieve temporal information fusion and spatial detail enhancement in the channel dimension.

[0014] Step 3 is implemented in the following steps: The low-resolution video set constructed As input to a deep neural network for video super-resolution, with a corresponding high-resolution video set. As labels, deep neural networks are optimized through end-to-end training to obtain a training model for video super-resolution reconstruction.

[0015] Step 4 is implemented in the following steps: In practical applications, the low-resolution video sequence to be processed is input into the deep neural network model trained in step 3, and the network output is the corresponding high-resolution video frame, thereby achieving super-resolution restoration of the input video sequence.

[0016] The beneficial effects of this invention are: (1) To address the problem of temporal flicker and lack of consistency caused by ignoring inter-frame correlation in single-frame super-resolution algorithms, this invention constructs a spatiotemporal convolutional attention module. By utilizing the synergistic effect of pseudo-3D static large kernel convolution and temporally aware dynamic convolution, rich local texture details are accurately restored while ensuring the consistency of the global structure of the video. The proposed method reduces the consumption of computing resources while improving performance. (2) This invention breaks through the limitations of traditional methods that rely on explicit motion estimation and compensation or high-overhead 3D convolution. By introducing a lightweight temporal shift operator and a trajectory-aware dynamic convolution mechanism, implicit multi-frame spatiotemporal alignment and interaction are achieved in the feature dimension. While fully exploring the complementary details between adjacent frames, the computational burden of the system is significantly reduced, providing the possibility for the application of video super-resolution algorithms in scenarios with limited computing power. (3) Compared with video super-resolution methods based on Transformer structure, this invention innovatively adopts a heterogeneous spatiotemporal modeling strategy, that is, long-range temporal dependency modeling is achieved through cross-dimensional tensor recombination in key layers to ensure the integration of global information; while the spatiotemporal convolutional attention module is used in other layers for feature enhancement. This hierarchical optimization design, while maintaining strong representation capabilities, greatly reduces the number of parameters and inference time, thus improving the efficiency of video super-resolution technology. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of the present invention; Figure 2 This is a schematic diagram of the spatiotemporal enhancement module constructed according to the present invention; Figure 3 This is a schematic diagram of the spatiotemporal convolutional attention module proposed in this invention; Figure 4 shows the low-resolution video frames and corresponding high-resolution video frames used when the upsampling factor is 4 in the simulation experiment of this invention. Figure 5 shows the video frames reconstructed by the image super-resolution method based on convolutional attention and the super-resolution reconstructed video frames when the upsampling factor is 4 in the simulation experiment of this invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] This invention presents a multi-frame collaborative reconstruction method for video super-resolution tasks. By modeling inter-frame temporal correlations in the feature domain, it achieves fine reconstruction of low-resolution video sequences into high-resolution videos. Specifically, this invention introduces a temporal shift mechanism and a trajectory-aware dynamic convolution strategy into the feature domain to complete multi-frame temporal information interaction and implicit alignment without explicit optical flow estimation, thereby significantly improving temporal fusion efficiency. Subsequently, a convolutional attention module combining large-kernel convolution and dynamic small-kernel convolution is employed to expand the receptive field while enhancing the representation of high-frequency textures and structural details, and effectively controlling computational complexity. Finally, using low-resolution video sequences as network input samples and corresponding high-resolution video frames as supervision labels, the proposed deep network model is trained in an end-to-end manner to achieve high-quality super-resolution reconstruction of the target video frames.

[0020] This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0021] In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0022] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0023] Step 2: Construct a deep neural network structure for video super-resolution; The deep neural network structure in step 2 is as follows: Figure 1 As shown, the overall network structure is as follows: First, the input low-resolution video sequence... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

[0024] Step 2 is implemented in the following steps: Step 2.1, Multi-frame Feature Extraction Module: Considering the advantages of feature representation in structural expression and detail reconstruction, the input low-resolution video sequence is first processed... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Each enhancement block in step 2.2 consists of two substructures cascaded in series. The feature map is first fed into the layer normalization and convolutional feedforward network, and then processed by the first substructure, i.e., the spatiotemporal window attention module, and the second substructure, i.e., the spatiotemporal convolutional attention module, in sequence. Finally, the feature map is transformed by the layer normalization and convolutional layer, and residual connection is made with the input to reconstruct the feature map. Spatiotemporal window attention module: This module receives the feature map processed by the normalization layer and convolutional feedforward network in step 2.2 as input. This input feature map contains the spatial local features and temporal series features of multiple frames of video. The module reconstructs these features into a unified vector space through spatiotemporal dimension projection technology, and uses the self-attention mechanism to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames. Spatiotemporal Convolutional Attention Module: This module receives feature maps from the preceding substructures in step 2.2 (such as the spatiotemporal window attention module) as input. First, it achieves information interaction between adjacent frames without increasing parameters through temporal shifting operations. Then, it splits the feature maps into a main branch and an auxiliary branch along the channel dimension. The main branch uses two parallel paths for feature extraction: one path uses the main branch features to generate a dynamic convolution kernel, and then implicitly aligns motion features and reduces dimensionality through depthwise convolution; the other path extracts static background information through shared large kernel convolution (kernel parameters are initialized by the network and dynamically updated), and fuses the temporal context through temporal aggregation operations. The two outputs are added together and concatenated with the auxiliary branch features to achieve temporal information fusion and spatial detail enhancement in the channel dimension.

[0025] Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced to accelerate network convergence and learn high-frequency residual information. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is finally output.

[0026] Step 3: Train the network based on paired video data Step 3 is implemented in the following steps: The low-resolution video set constructed As input to a deep neural network for video super-resolution, with a corresponding high-resolution video set. As labels, deep neural networks are optimized through end-to-end training to obtain a training model for video super-resolution reconstruction.

[0027] Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0028] Step 4 is implemented in the following steps: In practical applications, the low-resolution video sequence to be processed is input into the deep neural network model trained in step 3, and the network output is the corresponding high-resolution video frame, thereby achieving super-resolution restoration of the input video sequence.

[0029] Example 1 This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; Step 2: Construct a deep neural network structure for video super-resolution; Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0030] Example 2 This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0031] In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0032] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0033] Step 2: Construct a deep neural network structure for video super-resolution; Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0034] Example 3 This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0035] In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0036] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0037] Step 2: Construct a deep neural network structure for video super-resolution; The deep neural network structure in step 2 is as follows: Figure 1 As shown, the overall network structure is as follows: First, the input low-resolution video sequence... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

[0038] Step 2 is implemented in the following steps: Step 2.1, Multi-frame Feature Extraction Module: Considering the advantages of feature representation in structural expression and detail reconstruction, the input low-resolution video sequence is first processed... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced to accelerate network convergence and learn high-frequency residual information. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is finally output.

[0039] Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0040] Example 4 This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0041] In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0042] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0043] Step 2: Construct a deep neural network structure for video super-resolution; The deep neural network structure in step 2 is as follows: Figure 1 As shown, the overall network structure is as follows: First, the input low-resolution video sequence... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

[0044] Step 2 is implemented in the following steps: Step 2.1, Multi-frame Feature Extraction Module: Considering the advantages of feature representation in structural expression and detail reconstruction, the input low-resolution video sequence is first processed... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced to accelerate network convergence and learn high-frequency residual information. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is finally output.

[0045] Each enhancement block in step 2.2 consists of two substructures cascaded in series. The feature map is first fed into the layer normalization and convolutional feedforward network, and then processed by the first substructure, i.e., the spatiotemporal window attention module, and the second substructure, i.e., the spatiotemporal convolutional attention module, in sequence. Finally, the feature map is transformed by the layer normalization and convolutional layer, and residual connection is made with the input to reconstruct the feature map. Spatiotemporal window attention module: This module receives the feature map processed by the normalization layer and convolutional feedforward network in step 2.2 as input. This input feature map contains the spatial local features and temporal series features of multiple frames of video. The module reconstructs these features into a unified vector space through spatiotemporal dimension projection technology, and uses the self-attention mechanism to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames. Spatiotemporal Convolutional Attention Module: This module receives feature maps from the preceding substructures in step 2.2 (such as the spatiotemporal window attention module) as input. First, it achieves information interaction between adjacent frames without increasing parameters through temporal shifting operations. Then, it splits the feature maps into a main branch and an auxiliary branch along the channel dimension. The main branch uses two parallel paths for feature extraction: one path uses the main branch features to generate a dynamic convolution kernel, and then implicitly aligns motion features and reduces dimensionality through depthwise convolution; the other path extracts static background information through shared large kernel convolution (kernel parameters are initialized by the network and dynamically updated), and fuses the temporal context through temporal aggregation operations. The two outputs are added together and concatenated with the auxiliary branch features to achieve temporal information fusion and spatial detail enhancement in the channel dimension.

[0046] Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0047] Example 5 This invention presents a video super-resolution method based on spatiotemporal convolutional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1)It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

[0048] In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

[0049] Low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

[0050] Step 2: Construct a deep neural network structure for video super-resolution; The deep neural network structure in step 2 is as follows: Figure 1 As shown, the overall network structure is as follows: First, the input low-resolution video sequence... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

[0051] Step 2 is implemented in the following steps: Step 2.1, Multi-frame Feature Extraction Module: Considering the advantages of feature representation in structural expression and detail reconstruction, the input low-resolution video sequence is first processed... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Each enhancement block in step 2.2 consists of two substructures cascaded in series. The feature map is first fed into the layer normalization and convolutional feedforward network, and then processed by the first substructure, i.e., the spatiotemporal window attention module, and the second substructure, i.e., the spatiotemporal convolutional attention module, in sequence. Finally, the feature map is transformed by the layer normalization and convolutional layer, and residual connection is made with the input to reconstruct the feature map. Spatiotemporal window attention module: This module receives the feature map processed by the normalization layer and convolutional feedforward network in step 2.2 as input. This input feature map contains the spatial local features and temporal series features of multiple frames of video. The module reconstructs these features into a unified vector space through spatiotemporal dimension projection technology, and uses the self-attention mechanism to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames. Spatiotemporal Convolutional Attention Module: This module receives feature maps from the preceding substructures in step 2.2 (such as the spatiotemporal window attention module) as input. First, it achieves information interaction between adjacent frames without increasing parameters through temporal shifting operations. Then, it splits the feature maps into a main branch and an auxiliary branch along the channel dimension. The main branch uses two parallel paths for feature extraction: one path uses the main branch features to generate a dynamic convolution kernel, and then implicitly aligns motion features and reduces dimensionality through depthwise convolution; the other path extracts static background information through shared large kernel convolution (kernel parameters are initialized by the network and dynamically updated), and fuses the temporal context through temporal aggregation operations. The two outputs are added together and concatenated with the auxiliary branch features to achieve temporal information fusion and spatial detail enhancement in the channel dimension.

[0052] Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced to accelerate network convergence and learn high-frequency residual information. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is finally output.

[0053] Step 3: Train the network based on paired video data Step 3 is implemented in the following steps: The low-resolution video set constructed As input to a deep neural network for video super-resolution, with a corresponding high-resolution video set. As labels, deep neural networks are optimized through end-to-end training to obtain a training model for video super-resolution reconstruction.

[0054] Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

[0055] Example 6 Compared to model-based video super-resolution methods, deep neural network-based methods, while achieving significant improvements in reconstruction quality, currently mainstream algorithms rely excessively on stacking deep and complex networks to gain performance, leading to a substantial increase in model parameters and inference time. This makes them difficult to run on devices with limited computing power, restricting their widespread application in various fields. To overcome the performance-speed imbalance of existing methods, this invention proposes a video super-resolution method based on spatiotemporal convolutional attention, which significantly improves processing efficiency while ensuring image reconstruction accuracy. The specific process steps of this invention for video super-resolution are as follows: (1) For the given N A high-resolution video sequence, which is used as a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as Furthermore, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thus obtaining a low-resolution video set. Any low-resolution video sequence is represented as .

[0056] (2) Constructing a deep neural network structure for video super-resolution: (2a) Multi-frame feature extraction module: Considering the advantages of feature representation in structural expression and detail reconstruction, the input low-resolution video sequence is first processed using... Convolution is used to map and convert the original pixel-domain video frames into high-dimensional feature representations, forming an initial multi-frame feature sequence.

[0057] (2b) Deep Spatiotemporal Augmentation Module: This is the core of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies. Each augmentation block contains two key substructures: i. Spatiotemporal window attention module: Through spatiotemporal dimension projection technology, the spatial local features and temporal series features of multi-frame videos are reconstructed into a unified vector space. The self-attention mechanism is used to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames.

[0058] ii. Spatiotemporal Convolutional Attention Module: Introduces temporal shift operations to achieve zero-parameter interaction between adjacent frames; then, features are separated from the channel dimension, one path is combined with a trajectory-aware dynamic kernel, and content-adaptive dynamic weights are generated by calculating inter-frame difference features to implicitly align motion features, and global information of the static background is extracted using shared large kernel convolution to obtain the sum of the two parts of features, which is then concatenated with the features separated from the other path to achieve the fusion of temporal information and the enhancement of spatial detail information in the feature channel dimension.

[0059] (2c) Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames. This operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the texture and structural information that is most critical for reconstructing intermediate reference frames.

[0060] (2d) Super-resolution reconstruction module: The aggregated features are first passed through a... The convolutional layers are smoothed. Subsequently, global residual connections are introduced to accelerate network convergence and learn high-frequency residual information. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, ultimately outputting the super-resolution reconstruction result of the target video frame.

[0061] (3) Training the network based on paired video data: Let 2T+1=5, that is, take 5 consecutive video frames as a sequence. Select the Vimeo training dataset containing 64612 complete videos. Each video sequence includes 7 noisy video frames. Randomly select any 5 consecutive video frames as a video sequence. Use these high-resolution video sequences as the high-resolution video training set in the embodiments of the present invention. The resolution of each video frame is [resolution value missing]. Furthermore, bicubic interpolation is performed on each high-resolution video frame to obtain the corresponding low-resolution dataset. The upsampling factor set for the comparative experiment in this invention is 4. Then, a deep neural network model is constructed using the corresponding set for training, resulting in a trained network model.

[0062] (4) Video super-resolution reconstruction: Based on 7824 high-resolution video sequences in the Vimeo test set, bicubic interpolation is used to obtain the corresponding low-resolution video sequences. The low-resolution video sequences are input into the trained deep neural network, and the network output is the corresponding high-resolution video frame.

[0063] The technical effects of the present invention will be specifically illustrated through simulation experiments below: 1. Simulation conditions: 1) The programming platform used in the simulation experiment was JetBrains PyCharm; 2) The structure of the spatiotemporal enhancement module constructed in the simulation experiment is as follows: Figure 2 As shown; the structure of the spatiotemporal convolutional attention module is as follows Figure 3 As shown; 3) The low-resolution video frames used in the simulation experiment are shown in Figure 4(a), and the high-resolution video frames are shown in Figure 4(b). 4) In the simulation experiment, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used to evaluate the super-resolution performance. The peak signal-to-noise ratio (PSNR) was defined as: ,

[0064] in, This represents the mean square error of the super-resolution reconstructed image. , They are The mean, , They are variance yes covariance, , They are two constants.

[0065] Simulation content: The method of this invention and the existing image super-resolution method ESC based on convolutional attention are used to perform super-resolution reconstruction on low-resolution video sequences respectively. In order to achieve fair comparison, the upsampling factor is set to 4, and the number of input video frames for both methods is 5 frames. The super-resolution results of video frames of any sequence in the Vimeo test set are shown in Figure 5. The video frame recovered by the image super-resolution method ESC based on convolutional attention is shown in Figure 5(a), and the super-resolution video frame obtained by the present invention is shown in Figure 5(b). It can be seen that the video frame reconstructed by the method of the present invention has clearer texture details and higher recovery quality.

[0066] 2. Comparison of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) Using the Vimeo test set containing 7824 videos as test data, the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the video frames recovered by the existing convolutional attention-based image super-resolution method ESC and the method of this invention were calculated respectively. The results are shown in Table 1.

[0067] Table 1. Comparison of Peak Signal-to-Noise Ratio (PSNR) (dB) of ESC and the average results of SSIM (Structural Similarity SSIM) with the super-resolution (upsampling factor of 4) method of this invention.

[0068] As can be seen from the experimental results in Table 1, the average peak signal-to-noise ratio (PSNR) of the super-resolution reconstructed image using the method of the present invention is 1.30 dB higher than that of the ESC, thus improving the accuracy of super-resolution.

Claims

1. A video super-resolution method based on spatiotemporal convolutional attention, characterized in that, The specific steps are as follows: Step 1: Convert the input high-resolution video sequence into a low-resolution video sequence to obtain a low-resolution video set; Step 2: Construct a deep neural network structure for video super-resolution; Step 3: Train the network based on paired video data Step 4: Video super-resolution reconstruction, thereby achieving super-resolution restoration of the input video sequence.

2. The video super-resolution method based on spatiotemporal convolutional attention according to claim 1, characterized in that, In step 1, for the given N A set of high-resolution video sequences, which will be used to construct a high-resolution tag set. Each video sequence is composed of (2T+1) It consists of a series of time-sequential video frames, therefore any high-resolution video sequence can be represented as: ,in i and t These represent the indices of the video sequence and the video frame, respectively. Indicates the first i In the video sequence, the th t High-resolution video frames.

3. The video super-resolution method based on spatiotemporal convolutional attention according to claim 2, characterized in that, In step 1, the high-resolution video sequence is downsampled using bicubic interpolation to generate a low-resolution video sequence, thereby obtaining a low-resolution video set. Any low-resolution video sequence is represented as ,therefore Indicates the first i In the video sequence, the th t Low-resolution video frames.

4. The video super-resolution method based on spatiotemporal convolutional attention according to claim 3, characterized in that, The low-resolution video set in step 1 ,in, This indicates a 4x downsampling operation.

5. The video super-resolution method based on spatiotemporal convolutional attention according to claim 3, characterized in that, The deep neural network structure in step 2 is specifically as follows: First, the input low-resolution video sequence is processed... Sent by A multi-frame feature extraction module composed of convolutional layers obtains a high-dimensional feature map. This feature map is then processed sequentially through multiple cascaded spatiotemporal enhancement modules before being input into a temporal aggregation module. The convolutional layer produces a transformed feature map. Finally, through residual connections and an upsampling module, a high-resolution video frame is reconstructed.

6. The video super-resolution method based on spatiotemporal convolutional attention according to claim 5, characterized in that, Step 2 is implemented in the following steps: Step 2.1, Multi-frame feature extraction module: First, the input low-resolution video sequence... use Convolution maps video frames in pixel space. The features are converted into high-dimensional feature representations to form an initial multi-frame feature sequence. Step 2.2, Deep Spatiotemporal Augmentation Module: This is the core part of the network, consisting of multiple stacked spatiotemporal augmentation blocks, designed to simultaneously capture intra-frame spatial texture and inter-frame temporal dependencies; Step 2.3, Temporal Aggregation Module: After deep feature enhancement, a feature sequence containing rich spatiotemporal information is obtained. This module expands the time dimension and concatenates it with the channel dimension, utilizing... Convolution performs dimensionality reduction and depth fusion on features from multiple frames; this operation will... T The temporal features of the frames are aggregated into single-frame features, preserving the most critical texture and structural information for reconstructing intermediate reference frames; Step 2.4, Super-resolution Reconstruction Module: The aggregated features are first processed through a... The convolutional layers are smoothed, and then global residual connections are introduced so that the network only needs to learn high-frequency residual details. Finally, subpixel convolution operations are used to expand the spatial resolution of the feature map to the target size, and the super-resolution reconstruction result of the target video frame is finally output.

7. The video super-resolution method based on spatiotemporal convolutional attention according to claim 6, characterized in that, Each enhancement block in step 2.2 consists of two substructures cascaded in series. The feature map is first fed into the layer normalization and convolutional feedforward network, and then processed by the first substructure, i.e., the spatiotemporal window attention module, and the second substructure, i.e., the spatiotemporal convolutional attention module, in sequence. Finally, the feature map is transformed by the layer normalization and convolutional layer, and residual connection is made with the input to reconstruct the feature map. Spatiotemporal window attention module: This module receives the feature map processed by the normalization layer and convolutional feedforward network in step 2.2 as input. This input feature map contains the spatial local features and temporal series features of multiple frames of video. The module reconstructs these features into a unified vector space through spatiotemporal dimension projection technology, and uses the self-attention mechanism to extract spatiotemporal features with global perception capabilities, thereby establishing long-range spatiotemporal context associations across frames. Spatiotemporal Convolutional Attention Module: This module receives the feature map output from the preceding substructure in step 2.2 as input. First, it achieves information interaction between adjacent frames without increasing parameters through temporal shifting operations. Then, it splits the feature map into a main branch and an auxiliary branch along the channel dimension. The main branch uses two parallel paths for feature extraction: one path uses the main branch features to generate a dynamic convolution kernel, and then implicitly aligns motion features and reduces dimensionality through depthwise convolution; the other path extracts static background information through shared large kernel convolution, and fuses the temporal context through temporal aggregation operations. The two outputs are added together and concatenated with the auxiliary branch features to achieve temporal information fusion and spatial detail enhancement in the channel dimension.

8. The video super-resolution method based on spatiotemporal convolutional attention according to claim 7, characterized in that, Step 3 is implemented in the following steps: The low-resolution video set constructed As input to a deep neural network for video super-resolution, with a corresponding high-resolution video set. As labels, deep neural networks are optimized through end-to-end training to obtain a training model for video super-resolution reconstruction.

9. The video super-resolution method based on spatiotemporal convolutional attention according to claim 8, characterized in that, Step 4 is implemented in the following steps: In practical applications, the low-resolution video sequence to be processed is input into the deep neural network model trained in step 3, and the network output is the corresponding high-resolution video frame, thereby achieving super-resolution restoration of the input video sequence.

Images