Video super-resolution reconstruction method and system

Abstract

The present invention discloses a video super-resolution reconstruction method and system, which relate to the technical field of video restoration processing. The method comprises the following steps: collecting a video frame sequence to be reconstructed and a reference frame sequence; extracting a dense optical flow field between the reference frame and the target frame; geometrically aligning the target frame with the reference frame according to the dense optical flow field to obtain motion consistency features; extracting features from the video frame sequence through a parameterized residual scaling module; inputting the extracted features into a bidirectional propagation module to process forward and backward image sequences respectively, extracting temporal forward features and temporal reverse features, and then fusing them to obtain detail features; adaptively fusing the motion consistency features and the detail features, and feeding the fused features into a reconstruction network to obtain deep visual features; implementing pixel-level adaptive reconstruction according to the deep visual features to obtain a super-resolution video sequence; while maintaining feature space consistency, the method significantly improves the feature matching accuracy in complex motion scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of video restoration processing, and in particular to a video super-resolution reconstruction method and system. Background Art

[0002] Video super-resolution technology aims to reconstruct high-resolution videos by mining complementary information within video frame sequences. With the rapid adoption of ultra-high-definition display devices, users are increasingly demanding higher quality video content, while the resolution specifications of different display terminals vary significantly.

[0003] Traditional video super-resolution techniques typically build independent models for fixed scaling, making them difficult to dynamically adapt to diverse display requirements. This can lead to issues such as resolution mismatch, detail loss, and redundant computing resources in cross-device scenarios. Furthermore, a core challenge of traditional video super-resolution methods lies in the efficient alignment and fusion of spatiotemporal features across frames.

[0004] Existing techniques are primarily categorized into two paradigms: explicit and implicit alignment. Explicit alignment achieves pixel-level compensation through motion estimation, offering stable performance in scenes with large displacements. However, its performance is highly dependent on the accuracy of optical flow estimation, making it susceptible to compensation bias in occluded areas or scenes with weak motion. Implicit alignment, on the other hand, utilizes neural networks to autonomously learn inter-frame associations. While offering the advantage of dynamic adaptability, it faces bottlenecks such as insufficient interpretability and limited ability to model complex motion. In short, a single alignment approach struggles to adapt to diverse motion types and fails to balance reconstruction performance and robustness in complex scenes where rigid and non-rigid motion coexist. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, such as the single alignment method is difficult to adapt to different motion types and cannot balance reconstruction capability and robustness in complex scenes where rigid and non-rigid motion coexist, the present invention proposes a video super-resolution reconstruction method and system, constructs a dual-path architecture of explicit-implicit hybrid alignment, generates motion trajectory fields through a pre-trained optical flow network and uses a differentiable bilinear sampling algorithm to complete pixel-level motion compensation, and realizes feature alignment through the modeling capability of the spatiotemporal Transformer on a bidirectional propagation network, combined with a multi-scale pyramid fusion strategy, thereby greatly improving the problems existing in the existing technology.

[0006] A video super-resolution reconstruction method comprises the following steps:

[0007] Acquiring a target frame image and a center frame image in a video frame sequence to be reconstructed;

[0008] The dense optical flow field between the target frame image and the center frame image is extracted through the parameter-frozen optical flow prediction model. The target frame image is geometrically aligned with the center frame image according to the dense optical flow field. The aligned frame sequence is shallowly encoded to obtain motion consistency features.

[0009] The shallow visual features of the video frame sequence are extracted by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module; the extracted shallow visual features are input into a bidirectional propagation module based on a spatiotemporal Transformer to obtain temporal forward features and temporal reverse features; 3D position encoding is introduced into the spatiotemporal attention mechanism of the spatiotemporal Transformer; the temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detail features;

[0010] The motion consistency features and detail features are fused; hierarchical feature extraction is performed on the fused features to obtain deep visual features; pixel-level reconstruction is performed based on the deep visual features to generate the final super-resolution video sequence.

[0011] Furthermore, the target frame image is geometrically aligned with the center frame image according to the dense optical flow field, and shallow feature encoding is performed on the aligned frame sequence to obtain motion consistency features. Specifically, the following steps are included:

[0012] According to the dense optical flow field, a differentiable bilinear sampling algorithm is used to geometrically align the target frame image with the center frame image.

[0013] The aligned frame sequence is shallowly encoded through a lightweight convolutional network, specifically by upgrading the aligned frame sequence to a high-dimensional semantic feature space, compressing the upgraded multi-frame features into a single-frame representation vector using a temporal dimension adaptive average pooling operation, and implicitly completing the adaptive fusion of motion information through a dynamic weight allocation mechanism to obtain motion consistency features.

[0014] Furthermore, the shallow visual features of the video frame sequence are extracted by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module. Specifically, the method includes: using a serially cascaded five-layer stacking structure to extract features from local details to global semantics; dynamically adjusting the residual connection strength using learning parameters to adaptively adjust the feature fusion ratio, and eliminating the statistical distribution differences between different video frames by removing the BatchNorm layer; features at each level are extracted through progressive transmission, and each layer includes a 3×3 convolution layer, a LeakyReLU activation function, and a parameterized residual connection; the feature extraction process is expressed as:

[0015] ;

[0016] in, represents the learning parameters, and Indicates a convolution layer with a convolution kernel of 3*3. Represented as LeakyReLU activation function, represents the input feature map, Represents the output feature map.

[0017] Furthermore, the extracted shallow visual features are input into a bidirectional propagation module with a spatiotemporal Transformer as a basic unit to obtain temporal forward features and temporal reverse features, which specifically includes the following steps:

[0018] In the forward processing stage, shallow visual features are input into the forward processing network composed of stacked spatiotemporal Transformers in the original time sequence. The spatiotemporal attention mechanism with 3D position encoding is used to aggregate the shallow visual features of multiple frames in the video frame sequence, and the temporal forward features are extracted frame by frame.

[0019] In the reverse processing stage, the shallow visual features are mirror-flipped along the temporal dimension and fed into an independent reverse processing network. By introducing the spatiotemporal attention mechanism of 3D position encoding, the shallow visual features of multiple frames in the video frame sequence are aggregated, and the temporal forward features are extracted frame by frame.

[0020] Furthermore, the aggregating of multiple shallow visual features of frames in a video frame sequence by introducing a spatiotemporal attention mechanism of 3D position encoding specifically includes the following steps:

[0021] Generate 3D dynamic position encoding through 3D position convolution;

[0022] 3D dynamic position encoding and shallow visual features are segmented to obtain local blocks;

[0023] Generate the query matrix Q with the local block segmented by 3D dynamic position encoding as the core, and generate the key matrix K and value matrix V with the local block segmented by shallow visual features as the core;

[0024] The similarity matrix is ​​obtained by performing dot product calculation on the transpose of the query matrix Q and the key matrix K;

[0025] Perform weighted summation of the similarity matrix and the value matrix V to obtain the aggregated features;

[0026] The aggregated features are input into the attention mechanism module of Transformer, and the position-driven calculation strategy is used to achieve implicit inter-frame alignment and feature fusion.

[0027] Furthermore, the temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detailed features, which specifically includes the following steps:

[0028] Concatenate the time series forward features and time series reverse features according to the spatial dimension;

[0029] Using a progressive spatial-temporal downsampling strategy, the concatenated features are subjected to 3D average pooling operations with different downsampling coefficients to construct a feature pyramid.

[0030] The spatial-temporal dimension of feature pyramid cross-resolution features is aligned using a differentiable trilinear interpolation algorithm;

[0031] All aligned features are concatenated in the channel dimension and averaged to output detail features with detail sensitivity and context perception.

[0032] Furthermore, the motion consistency feature and the detail feature are fused using a dynamic gating fusion mechanism. Specifically, the following steps are included:

[0033] The motion consistency feature and detail feature are spliced ​​along the channel dimension to obtain the joint feature;

[0034] The weight of the joint features is estimated through the gating network composed of convolutional layers, and the Softmax normalization operation is performed along the channel dimension to obtain the final fusion features.

[0035] Furthermore, the fused features are subjected to hierarchical feature extraction to obtain deep visual features, which specifically includes the following steps:

[0036] Hierarchical feature extraction is performed on the fused features using stacked multiple residual group modules. Each residual group module consists of a residual channel attention block. Each residual channel attention block includes two convolutional layers and a channel attention module. The first convolutional layer is used to extract local features. In the channel attention module, global average pooling is used to compress the spatial information of local features. After learning the channel weights, the second convolutional layer uses the sigmoid activation function to generate a channel attention map of local features.

[0037] Multiply the channel attention map with the extracted local features channel by channel to complete the feature recalibration of the channel dimension;

[0038] The calibrated features are extracted through the convolution layer to obtain visual features, and the residual calculation of the visual features and the fused features is performed to obtain deep visual features.

[0039] Furthermore, the pixel-level reconstruction is performed based on the deep visual features to generate the final super-resolution video sequence; specifically, the following steps are included:

[0040] Generate content-aware weights of deep visual features based on dynamic convolution kernels;

[0041] Constructing a normalized coordinate network, generating extended sampling coordinates for each position by superimposing a predefined kernel offset; expanding the extended sampling coordinates to obtain a continuous tensor; the continuous tensor represents the neighborhood coordinates that can be sampled at each output position;

[0042] According to the continuous tensor, the deep visual features are used to obtain the neighborhood sampling features through the bilinear interpolation algorithm; and the neighborhood sampling features are rearranged to form a feature cube;

[0043] After aligning the content-aware weighted interpolation to the target resolution, Einstein summation is performed with the feature cube to complete neighborhood pixel fusion;

[0044] The residual calculation is performed between the features after neighborhood pixel fusion and the central frame image of the video sequence to obtain the reconstructed super-resolution video.

[0045] The present invention also includes a video super-resolution reconstruction system, comprising:

[0046] An acquisition module, used for acquiring a target frame image and a center frame image in a video frame sequence to be reconstructed;

[0047] The explicit alignment module is used to extract the dense optical flow field between the target frame image and the center frame image through the parameter-frozen optical flow prediction model, geometrically align the target frame image with the center frame image based on the dense optical flow field, and perform shallow feature encoding on the aligned frame sequence to obtain motion consistency features;

[0048] An implicit alignment module is configured to extract shallow visual features from a video frame sequence by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module. The extracted shallow visual features are input into a bidirectional propagation module based on a spatiotemporal Transformer to obtain temporal forward features and temporal reverse features. 3D position encoding is introduced into the spatiotemporal attention mechanism of the spatiotemporal Transformer. The temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detailed features.

[0049] The reconstruction module is used to fuse motion consistency features and detail features; perform hierarchical feature extraction on the fused features to obtain deep visual features; and perform pixel-level reconstruction based on the deep visual features to generate the final super-resolution video sequence.

[0050] The present invention provides a video super-resolution reconstruction method, which has the following beneficial effects:

[0051] Aiming at the inherent limitations of a single alignment method, the present invention constructs a dual-path architecture of explicit-implicit hybrid alignment; a dense optical flow field is generated through a parameter-frozen optical flow prediction model, and the target frame image is geometrically aligned with the center frame image to complete pixel-level motion compensation, and output alignment features with both high fidelity and spatiotemporal consistency, which significantly reduces motion blur and dislocation during feature fusion in dynamic scenes; feature alignment is achieved through the modeling capability of the spatiotemporal Transformer on a bidirectional propagation network, combined with a multi-scale pyramid fusion strategy. This bidirectional collaborative processing mechanism effectively breaks through the traditional single-path alignment by jointly modeling historical states and future trends. The proposed method addresses the limitations of the model's field of view in temporal modeling, significantly enhances the system's ability to represent complex spatiotemporal correlations, and introduces a spatiotemporal attention mechanism guided by spatiotemporal position encoding, effectively solving the attention ambiguity problem caused by the coupling of position and content features in traditional methods, and significantly improving the accuracy of multi-frame alignment. This method effectively solves the limitations of traditional single-path methods in motion blur and artifact suppression through feature fusion of different branches, and significantly improves the feature matching accuracy in complex motion scenes while maintaining the consistency of feature space. This solves the problem that existing technologies cannot balance reconstruction capability and robustness in complex scenes where rigid and non-rigid motion coexist. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is an architecture diagram of a method for super-resolution reconstruction of arbitrary-scale videos based on hybrid alignment in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the structure of a hybrid alignment module in an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of the structure of the explicit alignment module in an embodiment of the present invention;

[0055] Figure 4 This is a diagram of the implicit alignment module architecture in an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of bidirectional propagation in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of the spatiotemporal Transformer structure in an embodiment of the present invention;

[0058] Figure 7 Schematic diagram of the spatiotemporal attention mechanism structure in an embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of the structure of an image reconstruction module in an embodiment of the present invention;

[0060] Figure 9 Schematic diagram of the structure of the arbitrary scale upsampling module in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0062] The present invention proposes a video super-resolution reconstruction method; in view of the inherent limitations of a single alignment method, a dual-path architecture of explicit-implicit hybrid alignment is constructed. The explicit alignment branch generates a motion trajectory field through a pre-trained optical flow network and uses a differentiable bilinear sampling algorithm to complete pixel-level motion compensation, significantly enhancing the model's modeling ability for large displacement motion; the implicit alignment branch designs a spatiotemporal attention mechanism guided by spatiotemporal position encoding, and realizes feature alignment through the modeling ability of the spatiotemporal Transformer on a bidirectional propagation network, combined with a multi-scale pyramid fusion strategy. An adaptive gating network is designed to realize feature fusion of different branches. At the same time, a content-aware upsampling module based on a dynamic convolution kernel is designed to achieve upsampling reconstruction of any scale by generating pixel-level adaptive weight sums.

[0063] like Figure 1 As shown, the specific steps include:

[0064] S1. Collect continuous low-resolution video sequences , where the height and width are H×W.

[0065] In the hybrid alignment phase, a dual-path collaborative mechanism is employed: the explicit alignment branch generates motion trajectory fields using a pre-trained RAFT optical flow network and implements pixel-level motion compensation using differentiable deformation operations. The implicit alignment branch constructs a bidirectional temporal propagation network, captures spatiotemporal correlation features using a spatiotemporal attention mechanism based on 3D position encoding, and fuses bidirectional temporal features using a pyramid arbitrary-scale fusion strategy. The two features are adaptively fused via a gating network, where the gating unit dynamically assigns fusion weights to the explicit and implicit features using learnable parameters.

[0066] Temporal alignment in video super-resolution is the core link in solving the problems of motion blur and artifacts. The present invention proposes a hybrid alignment module (Hybrid Alignment Module, HAM), which effectively solves the limitations of traditional single-path methods in motion blur and artifact suppression by constructing a dual-path collaborative architecture of implicit alignment and explicit alignment, and combining it with a dynamic gating fusion mechanism. Specifically, the explicit alignment branch uses deformable convolution to achieve motion compensation, and the implicit alignment branch captures deep feature correlations through the attention mechanism, and finally the dynamic gating network realizes the adaptive fusion of different features. This hybrid paradigm significantly improves the feature matching accuracy in complex motion scenes while maintaining the consistency of the feature space, laying a reliable temporal alignment foundation for subsequent super-resolution reconstruction. The module structure is as follows Figure 2 shown.

[0067] (1) Explicit alignment module

[0068] The core principle of video super-resolution technology is to use multi-frame image data to reconstruct the high-frequency details missing in a single frame. The explicit alignment module achieves accurate matching between frames by establishing a motion compensation model based on optical flow estimation. Unlike traditional methods that rely on the limitations of optical flow estimation accuracy, the present invention introduces a RAFT network architecture based on deep learning. Its iterative correction mechanism significantly improves the robustness of motion vectors. RAFT (Recurrent All-Pairs Field Transforms) is an end-to-end optical flow prediction model, mainly used for pixel-level motion estimation between video frames. Specifically, the module obtains sub-pixel motion trajectories through a pre-trained spatiotemporal feature extractor, and uses a differentiable bilinear sampling algorithm to complete the feature field deformation, and finally outputs alignment features with both high fidelity and spatiotemporal consistency. Module details are as follows: Figure 3 shown.

[0069] The present invention uses a pre-trained optical flow estimation model to construct an explicit alignment module, and its core process is divided into two steps: first, the dense optical flow field between the reference frame and the target frame is extracted through the parameter-frozen RAFT-Small model, and a cross-frame pixel-level displacement mapping relationship is established; second, the target frame features are geometrically aligned to the reference frame coordinate system according to the optical flow field using a differentiable spatial transformation to eliminate inter-frame motion offsets. In this process, the optical flow prediction model uses parameter freezing as a priori motion sensor to avoid optimization conflicts introduced by updating optical flow weights during training. This explicit alignment strategy significantly reduces motion blur and dislocation during feature fusion in dynamic scenes through physically interpretable motion modeling, providing a geometrically consistent spatiotemporal representation basis for subsequent tasks.

[0070] The aligned frame sequence is encoded with a lightweight convolutional network to obtain motion consistency features. . In order to eliminate redundant temporal information, an adaptive average pooling operation in the temporal dimension is used to compress multi-frame features into a compact representation vector of a single frame, and the adaptive fusion of motion information is implicitly completed through a dynamic weight allocation mechanism. This architecture achieves a trade-off between computational efficiency and representation performance: on the one hand, the gradient transfer overhead is eliminated by freezing the optical flow estimation module; on the other hand, a lightweight feature extraction architecture is used to reduce the parameter requirement. This step follows the "alignment-fusion" hierarchical processing flow to ensure the interpretability and consistency of spatiotemporal features in a unified physical coordinate system, providing geometrically robust underlying feature representations for high-order visual tasks.

[0071] (2) Implicit alignment module

[0072] In video super-resolution reconstruction, the motion blur effect between adjacent frames significantly weakens the spatial consistency constraints between pixels. Traditional explicit motion compensation methods achieve inter-frame alignment through a motion estimation module, but their computational complexity is high and they lack robustness to fast-moving scenes. While implicit alignment methods avoid explicit motion calculations, the one-way propagation model is limited by its temporal modeling capabilities and struggles to capture the global spatiotemporal correlations of video sequences, leading to structural distortion and artifacts when reconstructing dynamic regions.

[0073] To address the above issues, we propose an arbitrary-scale feature fusion framework based on bidirectional spatiotemporal propagation. This framework effectively improves the super-resolution performance of dynamic scenes by constructing a closed-loop feature flow. The architecture consists of three core modules:

[0074] ① Feature Extractor: Robust extraction of video features is achieved through a parameterizable residual scaling strategy, enabling efficient feature abstraction from shallow textures to deep semantics. The feature extractor consists of five layers of improved residual feature extractors in series, each of which contains a 3×3 convolutional layer, a LeakyReLU activation function, and a parameterized residual connection. In the improved residual feature extractor, learnable parameters are used to α Dynamically adjust the residual connection strength, enabling the network to automatically adjust the feature fusion ratio based on the motion intensity of the video clip. By removing the BatchNorm layer from the standard residual extractor, statistical distribution differences between different video frames are eliminated, preventing feature distribution shifts. Through progressive feature transfer, feature extraction is completed, from local details to full-sentence semantics.

[0075] ② Bidirectional propagation network: Use spatiotemporal Transformer to process forward and backward image sequences respectively, and process temporal dependencies and reverse context information.

[0076] ③ Feature fusion module: It integrates bidirectional propagation features through multi-scale feature pyramid to achieve effective integration of global information. The overall architecture is as follows Figure 4 shown.

[0077] An improved residual feature extractor is used to process the input video sequence. First, a parameterized residual scaling module is constructed using learnable parameters. α Dynamically adjusting the residual connection strength enables the network to automatically adjust the feature fusion ratio based on the motion intensity of the video clip. By removing the BatchNorm layer, statistical distribution differences between different video frames are eliminated, preventing feature distribution shift. The network body adopts a five-layer stacked structure, each layer consisting of a 3×3 convolutional layer, a LeakyReLU activation function, and a parameterized residual connection. Features at each level are progressively transferred to complete feature extraction from local details to global semantics. The feature extractor process can be expressed as:

[0078] (1)

[0079] in, represents a learnable scaling factor, and Indicates a convolution layer with a convolution kernel of 3*3. Represented as LeakyReLU activation function. represents the input features, Represents the shallow visual features of the output. Shallow visual features refer to relatively basic visual features extracted from an image. These features are usually closer to the input image and contain more pixel points and detailed information. Shallow visual features mainly include fine-grained information such as color, texture, edges, and corners.

[0080] The extracted shallow visual features are then fed into a bidirectional propagation module based on the spatiotemporal Transformer as the basic unit to construct bidirectional spatiotemporal features and process temporal dependencies. In the forward processing stage, the video sequence is input into the forward processing network composed of stacked spatiotemporal Transformers in the original temporal order, and the temporal forward features are extracted frame by frame; in the reverse processing stage, the input sequence is mirror-flipped along the time dimension and fed into an independent reverse processing network. Future contextual information is captured through reverse temporal modeling, and finally feature alignment is achieved through secondary temporal flipping to obtain temporal reverse features. This bidirectional collaborative processing mechanism effectively breaks through the field of view limitations of traditional unidirectional models in temporal modeling by jointly modeling historical states and future trends, and significantly enhances the system's ability to represent complex spatiotemporal associations. Bidirectional propagation such as Figure 5 shown.

[0081] The forward propagation process is shown in formula (2):

[0082] (2)

[0083] in, Indicated by m It consists of cascaded spatiotemporal Transformer blocks, Indicates the positive time series feature.

[0084] The back propagation process is shown in formula (3):

[0085] (3)

[0086] in, Indicated by m A network composed of a cascade of spatiotemporal Transformers to model reverse video sequences. Represents the time series reverse feature. The structure of spatiotemporal Transformer is as follows Figure 6 shown.

[0087] The core design of the spatiotemporal Transformer architecture follows the basic paradigm proposed by Vit. However, to better capture the spatiotemporal relationships in video sequences, an attention mechanism based on 3D position encodings is proposed to optimize multi-frame video alignment performance. This attention mechanism, through the structural design of 3D position encodings and high-dimensional visual features, constructs a position-guided global spatiotemporal interaction module. First, dynamic position encodings are generated through 3D position convolutions. These 3D position encodings are then concatenated with shallow visual features in the channel dimension and fed into the attention mechanism module. Within the attention mechanism module, a position-driven computational strategy is employed to achieve implicit inter-frame alignment and feature fusion. Specifically, the 3D dynamic position encodings serve as the core to generate a query matrix (Query), encoding the spatiotemporal position information of pixels in the video sequence. The 3D dynamic position encodings are mapped to the query matrix through grouped convolutional layers, ensuring independent representation of position information across channels and avoiding interference from semantic features. Key and value matrices are generated based on visual features. The key matrix is ​​generated through grouped convolutions, while the value matrix is ​​generated through standard convolutions, preserving cross-channel semantic fusion and ensuring that the value matrix contains richer detailed information. The dot product of the query matrix Q and the transpose of the key matrix K is used to calculate the similarity matrix, which is then weighted and summed with the value matrix V to obtain the aggregated features. This decoupled design enables the network to actively query the content area to be aligned based on position when calculating attention, effectively solving the attention ambiguity caused by the coupling of position and content features in traditional methods, and significantly improving the accuracy of multi-frame alignment.

[0088] By dividing the Q, K, and V feature matrices into 4x4 local blocks, the original pixel-level attention mechanism is transformed into block-level similarity matching, reducing the computational complexity from the square of the number of pixels O(n²) to the square of the number of blocks, achieving a linear increase in computational efficiency. Subsequently, the block sequence is split into multiple attention heads, each of which independently calculates the correlation between blocks. Specific details are as follows Figure 7 shown.

[0089] Finally, the multi-scale pyramid fusion module is used to extract and fuse features in different time and space dimensions to enhance the model's ability to capture multi-scale information and obtain detailed features. Specifically, the temporal forward features and temporal reverse features are spliced ​​according to the spatial dimension, and the spliced ​​features are used to construct a feature pyramid through 3D average pooling operations with different downsampling coefficients according to the progressive spatial-temporal downsampling strategy; using the progressive spatial-temporal downsampling strategy, a feature pyramid is constructed through 3D average pooling operations with different downsampling coefficients, forming a multi-level feature expression covering microscopic motion details to macroscopic scene semantics; then the spatial-temporal dimension alignment of cross-resolution features is achieved through the differentiable trilinear interpolation algorithm, effectively eliminating the geometric deviation between multi-scale features; the features of all scales are spliced ​​in the channel dimension and the average is taken, and then the redundant information is compressed through a lightweight convolutional layer, and a fused feature vector with both detail sensitivity and context perception is output.

[0090] (3) Dynamic gated fusion

[0091] The present invention uses a dynamic gated fusion mechanism to achieve adaptive fusion of explicit and implicit alignment features. First, the motion consistency features output by the explicit alignment module are and the detailed features output by the implicit alignment module Splicing is performed along the channel dimension. Then, weight estimation is performed through the gating network composed of convolutional layers to obtain the full channel weight map , = + ; and perform Softmax normalization along the channel dimension to ensure that the sum of explicit-implicit weights is 1. The final fusion feature is shown in formula (4).

[0092] (4).

[0093] In the S3 and image reconstruction stages, stacked residual groups are used to implement cross-layer feature reuse and improve the ability to restore high-frequency details.

[0094] Hierarchical feature extraction is performed by stacking multiple residual group modules. A residual link mechanism of shared features is adopted, and the original fused features are weighted after each residual group module to form a cross-level feature reuse link, which effectively alleviates the gradient vanishing problem and enhances the transmission efficiency of low-level features. Each residual group is composed of a residual channel attention block, and each residual attention block is composed of a "convolution layer-channel attention block-convolution layer" in series. The convolution layer is responsible for extracting local features. The channel attention module compresses spatial information through global average pooling, and uses 1×1 convolution to learn channel weights, and finally generates a channel attention map through the Sigmoid activation function. The channel attention map is multiplied channel by channel with the extracted local features to complete the feature recalibration of the channel dimension. This mechanism can dynamically enhance the feature response of important channels and suppress redundant information. The calibrated features are extracted through the convolution layer to obtain visual features, and the visual features are residually calculated with the fused features to obtain deep visual features. The specific details are as follows. Figure 8 shown.

[0095] S4. Aiming at the limitations of the current video super-resolution algorithm with fixed magnification, the present invention also constructs an arbitrary scale upsampling module based on a dynamic convolution kernel; implements nonlinear mapping of the feature map output by the image reconstruction module through cascaded convolution kernels; constructs a normalized coordinate network, and Generate extended sampling coordinates by superimposing predefined kernel offsets; expand the extended sampling coordinates to obtain a continuous tensor; the continuous tensor represents the number of samples required for each output position neighborhood coordinates; based on the continuous tensor, neighborhood sampling features are obtained through differentiable bilinear interpolation, and the sampling results are rearranged to form a feature cube; after the dynamic convolution kernel weight W is interpolated and aligned to the target resolution, Einstein summation is performed with the feature cube to complete neighborhood pixel fusion and obtain the target resolution scale; while maintaining content perception, video super-resolution of different scales is achieved, significantly broadening the application scenarios of video super-resolution algorithms.

[0096] The arbitrary scale upsampling module generates content-aware weights based on dynamic convolution kernels, and realizes pixel-level adaptive reconstruction through geometrically precise neighborhood sampling and Einstein summation, supporting stepless scaling and enhancing spatial continuity. In order to achieve image reconstruction at any scale, an arbitrary scale upsampling module based on dynamic convolution kernel generation is proposed. The arbitrary scale upsampling module consists of three parts: kernel weight prediction, deformable neighborhood sampling and adaptive fusion. The specific module process is as follows Figure 9 shown.

[0097] First, the deep visual features output by the image reconstruction module , nonlinear mapping is achieved by cascading convolution kernels, and the calculation process is shown in formula (5):

[0098] (5)

[0099] in, and is a 3×3 convolutional layer, Represents the RELU activation function. Output tensor Each spatial position correspond The normalized weights of the convolution kernel reflect the local semantic relevance of the input features.

[0100] To achieve any target resolution Adaptation, building a normalized coordinate network , each coordinate position satisfies For each position By superimposing predefined nuclear offsets Generate extended sampling coordinates. The calculation process is shown in formula (6):

[0101] (6)

[0102] in, is the scaling factor related to the input resolution. Expand the extended sampling coordinates to obtain a continuous tensor, which represents the number of samples required for each output position. Neighborhood coordinates, the calculation process is shown in formula (7):

[0103] (7)

[0104] The neighborhood sampling features are obtained by differentiable bilinear interpolation, and the sampling results are rearranged to form a feature cube. The calculation process is shown in formula (8) and formula (9):

[0105] (8)

[0106] (9)

[0107] Next, content-adaptive fusion is performed. After the dynamic convolution kernel weight W is interpolated and aligned to the target resolution, Einstein summation is performed with the feature cube to complete the neighborhood pixel fusion and obtain the target resolution scale. The calculation process is shown in formula (10):

[0108] (10)

[0109] in, Represents element-wise multiplication of channel-space positions, and the sum operation is along Dimensional.

[0110] Finally, the residual calculation is performed between the features after neighborhood pixel fusion and the central frame image of the video sequence to reconstruct the final super-resolution video.

[0111] This process dynamically generates position-by-position convolution kernel weights by learning the spatial semantic dependencies of input features, achieving high-quality, content-aware image upsampling. In this process, the generation of dynamic convolution kernels and feature sampling operations are jointly optimized through end-to-end gradient propagation, enabling the model to autonomously explore the spatial correlation of input features and adaptively adjust pixel fusion strategies for different semantic regions. Compared with traditional fixed interpolation techniques, this module breaks through the inductive bias limitations of traditional fixed interpolation techniques, achieves adaptive reconstruction of image content in the frequency domain, and effectively suppresses aliasing distortion of high-frequency components. At the same time, the decoupling design of the weight generation network and the dynamic convolution operation gives the model compatibility with arbitrary target resolutions and supports non-integer rate upsampling tasks.

[0112] Experimental analysis:

[0113] Dataset: During training, we used Vimeo-90K as the training set for the network. Vimeo-90K is a widely used benchmark dataset for video super-resolution tasks. This training set contains 64,612 sets of high-resolution video image sequences, each containing 7 frames. Bicubic downsampling (BI) is used to generate low-resolution and high-resolution frame pairs that are strictly aligned in time and space.

[0114] To test network performance, we selected the Vid4 dataset, a commonly used test dataset for video super-resolution tasks. Vid4 contains four high-definition video clips (e.g., cityscapes, action figures), and is commonly used to evaluate algorithms' ability to handle blur, texture detail, and motion consistency.

[0115] During the training process, the input image is cropped into 64*64 size image blocks as input. The network optimizer uses the Adam optimizer, where and The decay rates are set to 0.9 and 0.999 respectively, and the initial learning rate is set to , and cooperate with the cosine annealing learning rate scheduling strategy to achieve dynamic attenuation during training.

[0116] The L1 norm was used as the error metric for the loss function, and the batch size was fixed at 4. For training on different datasets, the model was trained for 100 full training cycles to ensure sufficient convergence. PNSR and SSIM were used as evaluation metrics to assess the performance of the proposed model. During training, the input sequence was downsampled to arbitrary scales (×1.0–×4.0) using a dual-scaling algorithm to generate LR frames of varying resolutions. A new random scaling factor was applied to each batch of data fed into the network.

[0117] The loss function adopts the Charbonnier loss constraint network, and the calculation formula is shown in formula (11):

[0118] (11)

[0119] in, represents the image reconstruction result, represents a high-resolution image, , which is used to control the smoothness of the loss function near zero.

[0120] Experimental comparison:

[0121] The compared methods are mainly divided into two categories: (1) single-frame image super-resolution methods of arbitrary scale: Meta-SR, LIIF, ArbSR, and LTE; (2) video super-resolution methods: EDVR, BasicVSR, IconVSR, and BasicVSR++. The addition of the "BI" mark after the compared method indicates that the output result has been further processed to make the output adapt to the target resolution.

[0122] Table 1 Experimental results of super-resolution reconstruction under symmetric scaling factors

[0123]

[0124] Table 1 shows the experimental results of super-resolution reconstruction under symmetric scaling factors. The experimental results show that the proposed method comprehensively outperforms traditional single-frame arbitrary-scale super-resolution methods in terms of both PNSR and SSIM objective evaluation metrics. This is because single-frame arbitrary-scale super-resolution methods fail to effectively model the spatiotemporal correlations in video sequences, resulting in over-reliance on the local features of a single frame during reconstruction. This comparison confirms the critical role of spatiotemporal features in video super-resolution. The hybrid alignment module proposed in this paper, through a bidirectional propagation network, achieves collaborative modeling of spatiotemporal features, significantly improving the processing capabilities of video sequences. Experimental results compared with video super-resolution algorithms show that the proposed method achieves comparable super-resolution results under integer scaling factors in terms of PNSR and SSIM objective evaluation metrics. Super-resolution results under non-integer scaling factors are superior to those of other video super-resolution methods. This achievement is attributed to the proposed arbitrary-scale upsampling module, which achieves adaptive weighted representation based on content features through kernel weight prediction, deformable neighborhood sampling, and adaptive fusion, effectively adjusting the feature representation of image content under different scaling factors.

[0125] Table 2 Comparison results of video super-resolution reconstruction under asymmetric scaling factors

[0126]

[0127] Table 2 shows the comparative results of video super-resolution reconstruction under asymmetric scaling factors. Experimental data demonstrates that the proposed method exhibits optimal super-resolution performance under various scaling factors, demonstrating its strong spatial adaptability and detail reconstruction capabilities. Compared to traditional interpolation and single-frame arbitrary-scale super-resolution methods, it maintains excellent results in both extreme scaling and medium-span tasks, validating the effectiveness of the arbitrary-scale upsampling module and its universality in complex scaling scenarios.

[0128] Ablation experiments: Ablation experiments were conducted to verify the effectiveness of the proposed method. All ablation experiments were conducted on the Vid4 dataset. This paper primarily experimentally validates the effectiveness of the hybrid alignment module, bidirectional propagation mechanism, and adaptive upsampling network. The experimental results of these experiments are presented in Tables 3 to 5. In the tables, (√) indicates the presence of the module, and (×) indicates the absence of the module.

[0129] Verifying the Effectiveness of the Hybrid Alignment Module: To further analyze the functional roles of each component within the Hybrid Alignment Module, this experiment conducted ablation tests on three key modules within the Hybrid Alignment Module for horizontal comparison. The ablation results, tested under different symmetric scaling factors, are shown in Table 3. For the experiments on the dynamic gating network, fixed-weight fusion was used for comparison, with the weights of both the implicit and explicit alignment modules set to 0.5.

[0130] Table 3 Ablation experiment of hybrid alignment module

[0131]

[0132] The results of ablation experiments show that the sub-modules in the hybrid alignment module have clear complementarity in the super-resolution task of any scale. The explicit alignment module implements explicit spatial transformation modeling based on optical flow compensation, which can significantly improve the quality of structured feature reconstruction under low-magnification scaling factors. This may be because the information obtained at low resolution is relatively rich, and the optical flow field formed is more accurate; the implicit alignment module learns nonlinear mapping relationships through the spatiotemporal Transformer, and can effectively restore image quality under high-magnification scaling factors; the dynamic gating network adopts a context-aware weight distribution mechanism, and performs well in evaluation indicators in various scenarios, reflecting the advantages of the dynamic fusion of explicit and implicit features. The three work together to construct a hierarchical feature alignment system. Explicit alignment ensures spatial geometric consistency, implicit alignment enhances local detail representation, and the dynamic gating network uses a scene-adaptive feature fusion strategy to enable the model to maintain good performance under different scaling factors, verifying the effectiveness of the hybrid architecture design.

[0133] Verifying the effectiveness of the bidirectional propagation mechanism: To further analyze the effectiveness of the bidirectional propagation mechanism in the implicit alignment module, this experiment will conduct ablation experiments based on different propagation directions for horizontal comparison. The results of the ablation experiments will be tested under different symmetric scaling factors. The results of the ablation experiments are shown in Table 4.

[0134] Table 4 Ablation experiment of bidirectional propagation mechanism

[0135]

[0136] Experimental results show that in video super-resolution tasks, the bidirectional propagation mechanism, which fuses the temporal information of future and past frames through forward and backward propagation, significantly outperforms the unidirectional propagation mechanism in reconstructing images, achieving optimal performance in image quality evaluation under all scaling factors. The experimental results of backward propagation are significantly lower than those of forward propagation. This is because videos typically follow temporal causality, where past frames determine future frames. Forward propagation is more consistent with natural motion patterns, while backward propagation requires "reverse time sequence" modeling, which may introduce noise due to dynamic occlusion in future frames, resulting in a decrease in the quality of the reconstructed image.

[0137] Verification of the effectiveness of the spatiotemporal attention mechanism:

[0138] To further analyze the effectiveness of the spatiotemporal attention mechanism of the spatiotemporal Transformer, we compared it with conventional self-attention. The results of the ablation experiment were tested under different symmetric scaling factors. The results of the ablation experiment are shown in Table 5.

[0139] Experimental results show that the Spatiotemporal Transformer, by introducing a spatiotemporal attention mechanism based on 3D position encoding, significantly outperforms traditional self-attention models in video super-resolution tasks. Its PSNR and SSIM improve significantly at all scaling factors, with a particularly strong advantage at high magnification. The 3D encoding enables the model to implicitly align spatiotemporal information across frames and dynamically aggregate multi-frame features to compensate for motion blur and detail loss, thereby enhancing high-frequency detail recovery and structural coherence. This demonstrates the core value of spatiotemporal modeling for cross-frame feature fusion in video super-resolution tasks.

[0140] Table 5 Ablation experiment of spatiotemporal attention mechanism

[0141]

[0142] Based on the same inventive concept, the present invention also proposes a video super-resolution reconstruction system, comprising:

[0143] The acquisition module is used to acquire the target frame image and the center frame image in the video frame sequence to be reconstructed.

[0144] The explicit alignment module is used to extract the dense optical flow field between the target frame image and the center frame image through the parameter-frozen optical flow prediction model, geometrically align the target frame image with the center frame image according to the dense optical flow field, and perform shallow feature encoding on the aligned frame sequence to obtain motion consistency features.

[0145] The implicit alignment module is used to extract shallow visual features from video frame sequences by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module; the extracted shallow visual features are input into a bidirectional propagation module based on the spatiotemporal Transformer to obtain temporal forward features and temporal reverse features; 3D position encoding is introduced into the spatiotemporal attention mechanism of the spatiotemporal Transformer; the temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detail features.

[0146] The reconstruction module is used to fuse motion consistency features and detail features; perform hierarchical feature extraction on the fused features to obtain deep visual features; and perform pixel-level reconstruction based on the deep visual features to generate the final super-resolution video sequence.

[0147] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A video super-resolution reconstruction method, characterized in that: The following steps are involved: Acquiring a target frame image and a center frame image in a video frame sequence to be reconstructed; The dense optical flow field between the target frame image and the center frame image is extracted through the parameter-frozen optical flow prediction model. The target frame image is geometrically aligned with the center frame image according to the dense optical flow field. The aligned frame sequence is shallowly encoded to obtain motion consistency features. The shallow visual features of the video frame sequence are extracted by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module; the extracted shallow visual features are input into a bidirectional propagation module based on a spatiotemporal Transformer to obtain temporal forward features and temporal reverse features; 3D position encoding is introduced into the spatiotemporal attention mechanism of the spatiotemporal Transformer; the temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detail features; The motion consistency feature and the detail feature are fused; hierarchical feature extraction is performed on the fused features to obtain deep visual features; pixel-level reconstruction is performed based on the deep visual features to generate a final super-resolution video sequence; the pixel-level reconstruction based on the deep visual features to generate the final super-resolution video sequence specifically includes the following steps: content-aware weights of deep visual features are generated based on dynamic convolution kernels; a normalized coordinate network is constructed to generate extended sampling coordinates for each position by superimposing a predefined kernel offset; the extended sampling coordinates are expanded to obtain a continuous tensor; the continuous tensor represents the neighborhood coordinates that can be sampled at each output position; based on the continuous tensor, the deep visual features are subjected to a bilinear interpolation algorithm to obtain neighborhood sampling features; and the neighborhood sampling features are rearranged to form a feature cube; after the content-aware weights are interpolated and aligned to the target resolution, an Einstein sum calculation is performed with the feature cube to complete neighborhood pixel fusion; residual calculation is performed between the features after the neighborhood pixel fusion and the central frame image of the video sequence to obtain a reconstructed super-resolution video.

2. The video super-resolution reconstruction method according to claim 1, characterized in that: The method includes geometrically aligning the target frame image with the center frame image according to the dense optical flow field, performing shallow feature encoding on the aligned frame sequence, and obtaining motion consistency features. Specifically, the method includes the following steps: According to the dense optical flow field, a differentiable bilinear sampling algorithm is used to geometrically align the target frame image with the center frame image. The aligned frame sequence is shallowly encoded through a lightweight convolutional network, specifically by upgrading the aligned frame sequence to a high-dimensional semantic feature space, compressing the upgraded multi-frame features into a single-frame representation vector using a temporal dimension adaptive average pooling operation, and implicitly completing the adaptive fusion of motion information through a dynamic weight allocation mechanism to obtain motion consistency features.

3. The video super-resolution reconstruction method according to claim 1, characterized in that: The method extracts shallow visual features of video frame sequences by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module. Specifically, the method includes: extracting features from local details to global semantics from video frame sequences using a serially cascaded five-layer stacking structure; dynamically adjusting the residual connection strength using learning parameters to adaptively adjust the feature fusion ratio; and eliminating statistical distribution differences between different video frames by removing the BatchNorm layer; extracting features at each level through progressive propagation, with each layer including a 3×3 convolutional layer, a LeakyReLU activation function, and a parameterized residual connection. The feature extraction process is expressed as follows: ； in, represents the learning parameters, and Indicates a convolution layer with a convolution kernel of 3*3. Represented as LeakyReLU activation function, represents the input features, Represents the shallow visual features of the output.

4. The video super-resolution reconstruction method according to claim 3, wherein: The extracted shallow visual features are input into a bidirectional propagation module with a spatiotemporal Transformer as a basic unit to obtain temporal forward features and temporal reverse features, which specifically includes the following steps: In the forward processing stage, shallow visual features are input into the forward processing network composed of stacked spatiotemporal Transformers in the original time sequence. The spatiotemporal attention mechanism with 3D position encoding is used to aggregate the shallow visual features of multiple frames in the video frame sequence, and the temporal forward features are extracted frame by frame. In the reverse processing stage, the shallow visual features are mirror-flipped along the time dimension and fed into an independent reverse processing network. By introducing the spatiotemporal attention mechanism of 3D position encoding, the shallow visual features of multiple frames in the video frame sequence are aggregated, and the temporal reverse features are extracted frame by frame.

5. The video super-resolution reconstruction method according to claim 4, characterized in that: The method of aggregating multiple shallow visual features of a video frame sequence by introducing a spatiotemporal attention mechanism of 3D position encoding specifically includes the following steps: Generate 3D dynamic position encoding through 3D position convolution; 3D dynamic position encoding and shallow visual features are segmented to obtain local blocks; Generate the query matrix Q with the local block segmented by 3D dynamic position encoding as the core, and generate the key matrix K and value matrix V with the local block segmented by shallow visual features as the core; The similarity matrix is ​​obtained by performing dot product calculation on the transpose of the query matrix Q and the key matrix K; Perform weighted summation of the similarity matrix and the value matrix V to obtain the aggregated features; The aggregated features are input into the attention mechanism module of Transformer, and the position-driven calculation strategy is used to achieve implicit inter-frame alignment and feature fusion.

6. The video super-resolution reconstruction method according to claim 1, characterized in that: The time series forward features and time series reverse features are fused through a multi-scale feature pyramid to obtain detailed features. The specific steps include: Concatenate the time series forward features and time series reverse features according to the spatial dimension; Using a progressive spatial-temporal downsampling strategy, the concatenated features are subjected to 3D average pooling operations with different downsampling coefficients to construct a feature pyramid. The spatial-temporal dimension of feature pyramid cross-resolution features is aligned using a differentiable trilinear interpolation algorithm; All aligned features are concatenated in the channel dimension and averaged to output detail features with detail sensitivity and context perception.

7. The video super-resolution reconstruction method according to claim 1, characterized in that: The motion consistency feature and detail feature are fused using the dynamic gated fusion mechanism; specifically, the following steps are included: The motion consistency feature and detail feature are spliced ​​along the channel dimension to obtain the joint feature; The weight of the joint features is estimated through the gating network composed of convolutional layers, and the Softmax normalization operation is performed along the channel dimension to obtain the final fusion features.

8. The video super-resolution reconstruction method according to claim 1, characterized in that: The method performs hierarchical feature extraction on the fused features to obtain deep visual features, which specifically includes the following steps: Hierarchical feature extraction is performed on the fused features using stacked multiple residual group modules. Each residual group module consists of a residual channel attention block. Each residual channel attention block includes two convolutional layers and a channel attention module. The first convolutional layer is used to extract local features. In the channel attention module, global average pooling is used to compress the spatial information of local features. After learning the channel weights, the second convolutional layer uses the sigmoid activation function to generate a channel attention map of local features. Multiply the channel attention map with the extracted local features channel by channel to complete the feature recalibration of the channel dimension; The calibrated features are extracted through the convolution layer to obtain visual features, and the residual calculation of the visual features and the fused features is performed to obtain deep visual features.

9. A video super-resolution reconstruction system, characterized in that: include: An acquisition module, used for acquiring a target frame image and a center frame image in a video frame sequence to be reconstructed; The explicit alignment module is used to extract the dense optical flow field between the target frame image and the center frame image through the parameter-frozen optical flow prediction model, geometrically align the target frame image with the center frame image based on the dense optical flow field, and perform shallow feature encoding on the aligned frame sequence to obtain motion consistency features; An implicit alignment module is configured to extract shallow visual features from a video frame sequence by adaptively adjusting the feature fusion ratio through a parameterized residual scaling module. The extracted shallow visual features are input into a bidirectional propagation module based on a spatiotemporal Transformer to obtain temporal forward features and temporal reverse features. 3D position encoding is introduced into the spatiotemporal attention mechanism of the spatiotemporal Transformer. The temporal forward features and temporal reverse features are fused through a multi-scale feature pyramid to obtain detailed features. The reconstruction module is used to fuse motion consistency features and detail features; perform hierarchical feature extraction on the fused features to obtain deep visual features; perform pixel-level reconstruction based on the deep visual features to generate a final super-resolution video sequence; the pixel-level reconstruction based on the deep visual features to generate the final super-resolution video sequence specifically includes the following steps: generating content-aware weights of deep visual features based on dynamic convolution kernels; constructing a normalized coordinate network to generate extended sampling coordinates for each position by superimposing a predefined kernel offset; expanding the extended sampling coordinates to obtain a continuous tensor; the continuous tensor represents the neighborhood coordinates that can be sampled at each output position; based on the continuous tensor, obtain neighborhood sampling features from the deep visual features through a bilinear interpolation algorithm; and rearrange the neighborhood sampling features to form a feature cube; after the content-aware weights are interpolated and aligned to the target resolution, Einstein sum calculation is performed with the feature cube to complete neighborhood pixel fusion; residual calculation is performed between the features after neighborhood pixel fusion and the central frame image of the video sequence to obtain a reconstructed super-resolution video.

Images