Error accumulation and occlusion based arbitrary scale video super-resolution reconstruction method

Abstract

This invention belongs to the field of image super-resolution, and particularly to an arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion. The method specifically includes the following steps: preparing a dataset, constructing training data pairs, constructing a video super-resolution reconstruction network, designing a loss function and selecting evaluation metrics, model training and fine-tuning, and model saving and testing. The proposed robust arbitrary-scale video super-resolution reconstruction method constructs a unified framework that collaboratively enhances robustness through a flow-guided deformable alignment module and a bidirectional recurrent attention module. The flow-guided deformable alignment module integrates optical flow priors and learnable offsets, decomposing complex motion estimation into global coarse estimation and local fine-tuning, suppressing error input at its source. The bidirectional recurrent attention module collaboratively integrates the recursive characteristics of a bidirectional GRU and a multi-head cross-attention mechanism, constructing a feature propagation network that possesses both local temporal stability and global semantic awareness capabilities.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, specifically to an arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion. Background Technology

[0002] Video super-resolution reconstruction aims to reconstruct high-resolution video from low-resolution video sequences, and has wide application value in ultra-high-definition display, video surveillance, and other fields. With the development of 5G / 6G communication, users' requirements for video clarity are constantly increasing, and the demand for non-integer and asymmetric magnification in practical applications is growing, placing higher demands on the model's scale generalization ability and temporal consistency. In recent years, existing methods have initially achieved end-to-end arbitrary-scale video super-resolution and improved motion adaptability through techniques such as stream-guided deformable alignment. However, the implicit alignment mechanism of existing methods is still prone to failure when dealing with large displacements, non-rigid deformations, and occluded regions, leading to decreased reconstruction quality and temporal inconsistencies.

[0003] The Chinese authorization announcement number is "CN112580473B", and the name is "A Video Super-Resolution Reconstruction Method with Motion Features Fusion". This method first uses an adaptive discriminant subnet to determine whether the current frame is a keyframe. If it is, it is input into a high-precision keyframe reconstruction subnet based on depth back projection for fine reconstruction. If it is not a keyframe, the motion features between it and the neighboring keyframes are extracted and fused. Then, the reconstruction results of non-keyframes are quickly obtained through feature residual learning. Finally, the entire video is processed frame by frame to obtain the reconstructed video sequence. This method employs a divide-and-conquer strategy of "keyframes + non-keyframes," which balances accuracy and efficiency to some extent. However, the reconstruction quality of non-keyframes heavily relies on the feature accuracy of neighboring keyframes. When there is significant motion displacement or occlusion between keyframes and non-keyframes to be reconstructed, direct motion feature fusion cannot accurately compensate for inter-frame differences, leading to blurring or artifacts in the reconstruction results of non-keyframes. The motion feature extraction and fusion subnet for non-keyframes essentially transfers features from neighboring keyframes directly to the current frame after motion compensation. This unidirectional neighboring frame information transfer mechanism lacks effective suppression of error accumulation in the temporal propagation path. When processing long video sequences, the reconstruction error of non-keyframes amplifies with inter-frame propagation, and this method does not design a dedicated backpropagation mechanism. The method employs a recurring mechanism to block error propagation. However, its motion feature extraction approach is relatively basic, primarily estimating motion information through feature differences between adjacent frames. This approach lacks the ability to perform refined modeling of complex motion scenes (such as large displacements, non-rigid deformations, and occluded areas). While the upsampling module enhances feature representation through iterative feedback mechanisms of multiple upsampling and downsampling, it is essentially designed for tasks requiring predefined integer multiple magnification. It lacks corresponding processing mechanisms for non-integer or asymmetric scales commonly encountered in practical applications. Furthermore, the method relies on an adaptive discriminant subnet for distinguishing keyframes from non-keyframes, but the selection of the discrimination threshold and keyframe density affects the overall reconstruction quality. The method fails to provide an effective solution for dynamically balancing this trade-off under different video content.

[0004] In summary, designing a video super-resolution reconstruction method that can accurately model complex motion and occlusion scenes, effectively suppress the accumulation of long sequence errors, and support arbitrary scale scaling is an urgent problem to be solved in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion. This method solves the problems of existing arbitrary-scale video super-resolution methods, such as the easy failure of implicit alignment mechanisms when handling complex motion and occlusion scenes, severe accumulation of temporal errors in long sequence processing, and inability to adapt to the flexible requirements of non-integer and asymmetric scales.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution: An arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion includes the following steps: S1. Prepare the dataset: Prepare three publicly available video super-resolution benchmark datasets: Dataset 1 is the Vimeo-90K dataset; Dataset 2 is the UDM10 dataset; Dataset 3 is the Vid4 dataset.

[0007] S2, Constructing Training Data Pairs: High-resolution video frames are downsampled at arbitrary scales using a bicubic downsampling method to generate corresponding low-resolution video frame sequences, constructing input-output data pairs for network training. The scale factor can be set to integer, non-integer, or asymmetric values ​​to meet the training requirements of arbitrary-scale video super-resolution tasks. For non-integer or asymmetric scales, the boundaries of the high-resolution frames are first cropped to ensure that low-resolution frames of integer size are generated after downsampling, and then the corresponding low-resolution video frame sequences are generated through bicubic interpolation.

[0008] S3, Constructing a Video Super-Resolution Reconstruction Network: The network model includes a feature extraction and alignment module, a temporal feature propagation and enhancement module, and an arbitrary-scale reconstruction module. The feature extraction and alignment module uses shallow convolutions with shared weights to extract initial features from the input video sequence, and uses a flow-guided deformable alignment module to spatially align features of adjacent frames. The temporal feature propagation and enhancement module inputs the aligned features into a bidirectional recurrent attention module, and constructs a feature propagation network with both local temporal stability and global semantic awareness by collaboratively fusing the recursive characteristics of bidirectional GRUs and a multi-head cross-attention mechanism. The arbitrary-scale reconstruction module sequentially passes the enhanced temporal features through residual block cascade fusion and a spatiotemporally adaptive arbitrary-scale upsampling module to generate the final high-resolution video frame sequence based on the target scale factor. S4. Selection of loss function and design of evaluation metrics: The total loss function is a weighted sum of Charbonnier loss and temporal gradient consistency loss; the evaluation metrics are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to verify the super-resolution performance and generalization ability of the model. S5, Model Training and Fine-tuning: The preprocessed training data from step S1 is input into step S2 to generate a low-resolution video frame sequence of the corresponding scale to construct training data pairs, which are then fed into the network model constructed in step S3 for training. During training, the composite loss function designed in step S4 is used to calculate the loss value between the reconstruction result and the real high-resolution frame, and an optimizer is used for iterative parameter optimization. After training, the model is fine-tuned using the test dataset from step S1, and the model parameters are further optimized using a smaller learning rate to suppress the accumulation of temporal errors in long sequence processing and improve the image reconstruction quality in occluded areas and complex motion scenes. S6, Model Saving and Testing: After training and fine-tuning in step S5, solidify the fine-tuned network parameters and save the final small object detection model; use the remaining real datasets 2 and 3 from step S1 to test the model. Multiple consecutive images of any scale can be directly input into the trained end-to-end model to obtain the reconstructed result image.

[0009] Furthermore, in S1, dataset one is the Vimeo-90K dataset, which contains 91,701 seven-frame consecutive image sequences, each with a resolution of 448×256 pixels, and is used for network training; dataset two is the UDM10 dataset, which contains 10 video sequences covering various complex motion patterns and scene types; and dataset three is the Vid4 dataset, which contains 4 video sequences with an original image resolution of 720×576 pixels, and is used to verify the model's generalization ability in real-world scenarios. Furthermore, the bicubic downsampling method specifically refers to: for high-resolution images The pixels in the image are calculated using a bicubic interpolation function in the low-resolution image. For the pixel value at the corresponding position, the bicubic interpolation function uses the following kernel function: ; in, The distance between the point to be interpolated and its neighboring pixels is a parameter. The value is -0.5 or -1; the gray value of the target pixel in the low-resolution image is obtained by weighted summation of the gray values ​​of the pixels in the surrounding 4×4 neighborhood in the high-resolution image using the kernel function mentioned above.

[0010] Furthermore, in the feature extraction and alignment module of S3, the initial features of the input video sequence are extracted using shallow convolution with shared weights, and then the current frame features and the adjacent frame features are guided to the deformable alignment module for spatial alignment. The temporal feature propagation and enhancement module inputs the aligned features into the bidirectional recurrent attention module. By synergistically fusing the recursive characteristics of the bidirectional GRU and the multi-head cross attention mechanism, a feature propagation network with both local temporal stability and global semantic awareness is constructed. The arbitrary scale reconstruction module inputs the enhanced temporal features into the spatiotemporal adaptive arbitrary scale upsampling module after residual block cascading fusion, and generates the final high-resolution video frame sequence according to the target scale factor.

[0011] Furthermore, the feature extraction and alignment module extracts initial features from the input video sequence using shallow convolutions with shared weights, and spatially aligns the features of adjacent frames using a stream-guided deformable alignment module, given the input low-resolution video sequence. A total of 2N+1 consecutive images are generated. Initial features for each frame are first extracted using shallow convolutional layers with shared weights. Its expression is: ; in, This indicates a shallow convolution operation. This is the middle frame in a 2N+1 frame sequence. Then, the current frame features are... Features of adjacent frames The input stream guides the Flow-Guided Deformable Alignment (FGDA) module to perform spatial alignment. The alignment process can be represented as follows: ; in, For the features of the current frame, Features of adjacent frames A global mapping function for flow-guided deformable alignment modules; The flow-guided deformable alignment module consists of an offset prediction network, an optical flow estimation network, and deformable convolutions. The offset prediction network takes the channel concatenation of features from adjacent frames as input to predict the initial offset. Simultaneously, the SPyNet optical flow network is used to obtain the original image frames. and As input, predict the fundamental optical flow field. It provides global motion priors. The optical flow field and residual migration are fused point-by-point to obtain the final migration field. Then, adaptive alignment is performed using deformable convolution: ; in, To output the current pixel position on the feature map, The inherent first convolution kernel Each sampling offset, The weights of the k-th sampling point in deformable convolution are... The total offset field learned is in the th... Additional offset at each sampling point Original adjacent frame features; Aligned features from adjacent frames are concatenated with the current frame features along the channel dimension and then fed into a backend network that includes scale-adaptive convolution (Omni-Scale Convolution, OSConv). According to the target scale factor Perform feature enhancement and output the enhanced feature sequence. : ; in, These are the features of all frames after being aligned by the stream-guided deformable convolutional enhancement alignment module. It is the target scale factor. It is a backend network that includes scale-adaptive convolutions (OSConv); The temporal feature propagation and enhancement module will align the enhanced features. Input a bidirectional recurrent attention module to construct a feature propagation network that combines local temporal stability with global semantic awareness; The bidirectional cyclic attention module consists of three parts: a forward cyclic attention unit, a backward cyclic attention unit, and a bidirectional cross attention unit. The forward recurrent attention unit processes the sequence sequentially at the current time step in the forward time sequence. The input is the enhancement features of the current frame. Compared to the previous hidden state First, the current input and historical memory are fused through the gated recurrent unit (GRU). This process can be represented as follows: ; in, for The updated hidden state of the cell output at the current time step t. Enhance features for the current frame. This is the hidden state from the previous moment. For gated loop unit; Subsequently, by utilizing the cross-attention of multiple heads, For querying, use the historical hidden state sequence Perform a global context search for the key and value: ; in, The global context features are output by the multi-head cross-attention module. For multi-head cross-attention ensemble functions, Indicates from the first Frame to the The historical hidden state sequence of the frame, where N is the temporal window size; Finally, feature refinement is performed using residual connections and a multilayer perceptron, and the current hidden state is updated: ; in, The final refined feature output by the forward unit at time step t. This indicates that a normalization operation is performed after the residual connection. It is a multilayer perceptron. Indicates will The output hidden state is residually connected to the global context features output by multi-head cross-attention; Backward loop attention: The backward loop attention unit processes symmetric data, models the time sequence in reverse, and outputs backward features that contain future contextual information. ; The bidirectional cross-attention units perform forward-to-backward and backward-to-forward cross-attention operations respectively: ; ; in, This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention. This indicates that a normalization operation is performed after the residual connection. This indicates multiple points of cross-attention. The forward hidden features are the output of the forward recurrent attention unit. The backward hidden features are the output of the backward recurrent attention unit; The results from both approaches are then weighted and summed, and further fused using a multi-head self-attention layer and a multilayer perceptron (MLP) to output the final temporal features rich in global context. : ; in, The final output is a time-series feature rich in global context. This represents residual connection and layer normalization operations. It is a multilayer perceptron. This indicates multiple points of cross-attention. This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention; By synergistically integrating the recursive characteristics of bidirectional GRU with a multi-head cross-attention mechanism, a feature propagation network with both local temporal stability and global semantic awareness is constructed, which significantly enhances robustness to occluded regions and complex motion scenes. The arbitrary scale reconstruction module further fuses the enhanced temporal feature sequence through residual block concatenation, and then inputs it into the spatiotemporal adaptive arbitrary scale upsampling module to generate the final high-resolution video frame sequence according to the target scale factor. The spatiotemporal adaptive arbitrary-scale upsampling module includes a spatiotemporal filter branch and a scale filter branch. It dynamically generates upsampling convolution kernels by fusing scale information and spatiotemporal features, mapping low-resolution features to a high-resolution space. This process can be represented as: ; in, This represents the target pixel location on the high-resolution feature map. for The corresponding projection position on the low-resolution feature map For local domain windows; The spatiotemporal adaptive convolution weights are generated by the spatiotemporal filter branch. The scale-adaptive convolution weights (dependent on the target scale factor s) are generated by the scale filter branch. For learnable offsets, The feature value at the sampling point; The spatiotemporal filter branch generates spatiotemporal adaptive convolution weights using the enhanced temporal feature sequence as input. The scaling filter branch generates scale-adaptive modulation weights using the target scaling factor s as input. Finally, the two are multiplied point by point to form a joint upsampling convolution kernel, which outputs a high-resolution video frame sequence.

[0012] Furthermore, in the design of the evaluation metrics, the S4 loss function uses a weighted sum of Chabonie loss and temporal gradient consistency loss as the total loss function; the evaluation metrics use peak signal-to-noise ratio and structural similarity. The Charbonnier Loss is used to measure the pixel-level difference between the reconstructed high-resolution frame and the real high-resolution frame, ensuring the reconstruction accuracy of a single frame image. The temporal gradient consistency loss is used to constrain the inter-frame differences of the super-resolution sequence to remain consistent with the real sequence, thereby improving the temporal coherence and visual smoothness of the reconstructed video. The evaluation metrics used are peak signal-to-noise ratio and structural similarity to verify the super-resolution performance and generalization ability of the model. All metrics are calculated in the luminance channel.

[0013] Furthermore, the Chabonnier loss is defined as: ; Where T is the length of the video segment used during training. The total number of pixels in the image. and Let be the values ​​of the generated image and the real image at the i-th pixel, respectively, in frame t. It is the stability constant; The temporal gradient consistency loss is defined as follows: ; in, and Let be the generated image and the real image of frame t, respectively. express Norm, used to calculate the sum of the absolute values ​​of inter-frame difference errors; Finally, a balance coefficient is introduced. To balance the ratio between the two loss functions, a total loss function is formed: ; The peak signal-to-noise ratio (PSNR) is based on the pixel error between the reconstructed image and the real image, and is used to objectively measure the reconstruction of the image. Its definition is based on the mean square error (MSE). ; ; Where N represents the total number of pixels in the image, and i is the pixel index. Let be the pixel value at the i-th pixel in the super-resolution reconstructed image of frame t. Let L be the pixel value of the real high-resolution image in frame t at the i-th pixel, and L be the dynamic range of the pixel value, which is 255 for an 8-bit image. Structural similarity comprehensively evaluates the similarity between images from three dimensions: brightness, contrast, and structure, and is used to measure the ability of reconstructed images to preserve structure. ; in, y and x are the mean values ​​of the image, respectively. For variance, For covariance, To maintain a stable constant.

[0014] Compared with existing technologies, this invention provides a method for arbitrary-scale video super-resolution reconstruction based on error accumulation and occlusion, which has the following advantages: (1) This invention designs a novel end-to-end arbitrary scale video super-resolution reconstruction network and constructs a three-stage collaborative architecture of feature extraction and alignment, temporal feature propagation and enhancement, and arbitrary scale reconstruction. Through the organic integration of the flow-guided deformable alignment module and the bidirectional recurrent attention module, experiments show that the network architecture proposed in this invention achieves the best performance in multiple symmetric and asymmetric scale tasks on the Vid4 and UDM10 datasets. The quantitative indicators are significantly better than existing methods, and the qualitative results further verify its superiority in complex motion and occlusion scenarios.

[0015] (2) This invention designs a flow-guided deformable alignment module. By fusing optical flow priors and learnable offsets, the complex motion estimation is decomposed into two stages: global coarse estimation and local fine adjustment. Experiments show that the module significantly improves PSNR compared to the implicit alignment method on the Vid4 dataset symmetric scale task, effectively mitigating the error accumulation problem in the long time-series propagation process from the source.

[0016] (3) This invention designs a bidirectional cyclic attention module. By integrating the recursive characteristics of bidirectional GRU with the multi-head cross attention mechanism, a feature propagation network with both local temporal stability and global semantic perception capability is constructed. Experiments show that the module significantly improves the PSNR index of ordinary residual blocks on the asymmetric scale task of UDM10 dataset, effectively suppressing the continuous spread of error in the cyclic path.

[0017] (4) The video super-resolution model proposed in this invention has shown excellent performance in multiple symmetric and asymmetric scale tasks on the Vid4 and UDM10 datasets. The quantitative evaluation index has been significantly improved compared with the existing methods. The qualitative results show clear detail recovery ability and good temporal consistency in complex motion and occlusion scenes, indicating that the method proposed in this invention has strong robustness and generalization ability and can adapt to video super-resolution tasks of multiple scales and complex scenes. Attached Figure Description

[0018] Figure 1 This is a flowchart of an arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion. Figure 2 The overall framework diagram of the arbitrary-scale video super-resolution reconstruction method provided by the present invention; Figure 3 Network architecture diagram of the flow-guided deformable alignment module provided by the present invention; Figure 4 This invention provides a network architecture diagram for a bidirectional recurrent attention unit. Figure 5 This is a qualitative comparison diagram of the arbitrary-scale video super-resolution reconstruction method described in this invention and existing methods. Figure 6 This is a schematic diagram illustrating a quantitative comparison between the arbitrary-scale video super-resolution reconstruction method described in this invention and existing methods. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0020] A flowchart of an arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion is shown below. Figure 1 As shown, the method specifically includes the following steps: S1, Prepare the dataset: Prepare dataset one for network training, and train the entire reconstructed network.

[0021] Dataset 1 is the Vimeo-90K dataset, which contains 89,800 video sequences. From this dataset, 91,701 seven-frame consecutive image sequences were extracted as training data. Each sequence consists of seven consecutive images with a resolution of 448×256 pixels. A bicubic downsampling method was used to perform arbitrary-scale degradation on the training images, generating low-resolution frames corresponding to different resolutions. For non-integer scales, frame boundaries needed to be cropped to ensure that fractional resolution was not introduced during downsampling. For example, when downsampling a 576×704 pixel frame by a factor of 2.5, it needed to be cropped to 575×700 pixels to generate a low-resolution frame with an integer resolution of 230×280 pixels. Dataset 2 is the UDM10 dataset, which contains 10 video sequences covering various complex motion patterns and scene types, including representative dynamic scenes such as people walking, vehicles driving, and natural landscapes. It particularly includes a large amount of challenging video content with non-rigid deformation and occlusion areas. This dataset is often used to evaluate the reconstruction performance of models in complex motion scenes, especially their robustness in handling large displacement movements and occluded regions. Dataset 3 is the Vid4 dataset, which contains four video sequences: "city," "walk," "calendar," and "foliage." The original image resolution is 720×576 pixels, covering diverse scenes such as urban architecture, pedestrian movement, text details, and dense textures. Figure 5As shown in the qualitative experiment figures, input image one is a frame from the "foliage" sequence, and input image two is a frame from the "city" sequence. This dataset is often used to verify the model's generalization ability in real-world scenes, especially its performance in handling high-frequency texture details, edge preservation, and temporal consistency. S2, Constructing Training Data Pairs: Training data pairs are constructed using bicubic downsampling to simulate the resolution degradation process in real-world scenarios; bicubic downsampling uses high-resolution video frames as input, and adjusts the data according to the target scale factor. A downsampling operation is performed to generate corresponding low-resolution video frames. The scale factor is... and These represent the scaling ratios in the height and width directions, respectively. They can be set to integers (such as ×2, ×3, ×4), non-integers (such as ×1.5, ×2.25), or asymmetric values ​​(such as ×1.6 in the horizontal direction and ×3.05 in the vertical direction) to meet the training requirements of video super-resolution tasks of any scale. In the specific implementation process, for a given high-resolution video frame and target scale factor First, generate a sequence of 2N+1 consecutive low-resolution images. When dealing with non-integer or asymmetric scales, the boundaries of high-resolution frames need to be appropriately cropped to ensure that the low-resolution frames generated after downsampling have integer sizes. For example, for a high-resolution frame with an original resolution of 576×704, if a ×2.5 downsampling is required, the high-resolution frame should first be cropped to 575×700 pixels, and then a low-resolution frame with a size of 230×280 pixels should be generated through bicubic downsampling. Bicubic downsampling uses 16 pixels in a 4×4 neighborhood around the target pixel for weighted interpolation. By fitting the sampling function with a cubic polynomial, block artifacts are effectively suppressed while preserving image edge information, which can better simulate the resolution degradation process in real scenes. This degradation model dynamically generates low-resolution frames of arbitrary scales during the training phase, enabling the network to learn the mapping relationship of reconstructing high-resolution video frames from low-resolution inputs of different scales and proportions, thereby enhancing the model's adaptability to diverse scale factors. S3, Constructing a video super-resolution reconstruction network: The network model includes a feature extraction and alignment module, a temporal feature propagation and enhancement module, and an arbitrary scale reconstruction module; The feature extraction and alignment module extracts initial features from the input video sequence using shallow convolutions with shared weights, and spatially aligns features from adjacent frames using a stream-guided deformable alignment module. Given a low-resolution input video sequence... A total of 2N+1 consecutive images are generated. Initial features for each frame are first extracted using shallow convolutional layers with shared weights. Its expression is: ; in, This indicates a shallow convolution operation. This is the middle frame in a 2N+1 frame sequence. Then, the current frame features are... Features of adjacent frames The input stream guides the Flow-Guided Deformable Alignment (FGDA) module to perform spatial alignment. The alignment process can be represented as follows: ; in, For the features of the current frame, Features of adjacent frames A global mapping function for flow-guided deformable alignment modules; The flow-guided deformable alignment module consists of an offset prediction network, an optical flow estimation network, and deformable convolutions. The offset prediction network concatenates features from adjacent frames via channels. As input, the initial offset is predicted using a lightweight convolutional network. The value is the deformable convolution kernel size, and 2 corresponds to the two-dimensional spatial offset. Simultaneously, the SPyNet optical flow network is employed to utilize features from adjacent original image frames. and As input, predict the fundamental optical flow field. It provides robust global motion priors, expands the optical flow field dimension, and fuses it point-by-point with the learned residual offset to form the final offset field. This operation can be described as follows: ;in, This indicates the optical flow field of the prediction basis. This represents the initial offset for prediction. This means that a copy of the two-dimensional optical flow vector at each location is added point by point to the residual offset.

[0022] Finally, deformable convolution is used to perform adaptive feature sampling and alignment based on the fused offset field. Its mathematical expression can be: ; in, To output the current pixel position on the feature map, The inherent first convolution kernel Each sampling offset, The weights of the k-th sampling point in deformable convolution are... The total offset field learned is in the th... Additional offsets at each sampling point. This mechanism allows the network to dynamically adjust the sampling position based on the learned offsets, thereby accurately modeling non-rigid deformations and complex motions, providing a high-precision aligned feature base for subsequent temporal propagation.

[0023] The aligned features of adjacent frames are concatenated with the features of the current frame along the channel dimension and then fed into a backend network that includes scale-adaptive convolution (OSConv). According to the target scale factor Perform feature enhancement and output the enhanced feature sequence. : ; in, These are the features of all frames after being aligned by the stream-guided deformable convolutional enhancement alignment module. It is the target scale factor. It is a backend network that includes scale-adaptive convolutions (OSConv); The temporal feature propagation and enhancement module will align the enhanced features. The input is a Bidirectional Recurrent Attention Block (BRB), which consists of a forward recurrent attention unit, a backward recurrent attention unit, and a bidirectional cross attention unit. In the forward recurrent attention unit, features are enhanced based on the current frame during the forward temporal sequence. Compared to the previous hidden state For input, the current input is first fused with historical memory using GRU: ; in, for The updated hidden state of the cell output at the current time step t. Enhance features for the current frame. This is the hidden state from the previous moment. For gated loop unit; GRU provides fundamental stability for propagation by adaptively controlling the information flow through update and reset gates. Subsequently, it utilizes multi-head cross-attention to... For querying, use the historical hidden state sequence A global context search is performed on the key-value pair. The search process is as follows: ; in, The global context features are output by the multi-head cross-attention module. For multi-head cross-attention ensemble functions, Indicates from the first Frame to the The historical hidden state sequence of the frame, where N is the temporal window size; Finally, feature refinement is performed using residual connections and a multilayer perceptron, and the current hidden state is updated: ; in, The final refined feature output by the forward unit at time step t. This indicates that a normalization operation is performed after the residual connection. It is a multilayer perceptron. Indicates will The output hidden state is residually connected to the global context features output by multi-head cross-attention.

[0024] The backward recurrent attention unit processing mechanism is symmetrical, models the time sequence in reverse, and outputs backward features that contain future contextual information. ; The bidirectional cross-attention units perform forward-to-backward and backward-to-forward cross-attention operations respectively: ; ; in, This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention. This indicates that a normalization operation is performed after the residual connection. This indicates multiple points of cross-attention. The forward hidden features are the output of the forward recurrent attention unit. The backward hidden features are the output of the backward recurrent attention unit; The results from both approaches are then weighted and summed, and further fused using a multi-head self-attention layer and a multilayer perceptron (MLP) to output the final temporal features rich in global context. : ; in, The final output is a time-series feature rich in global context. This represents residual connection and layer normalization operations. It is a multilayer perceptron. This indicates multiple points of cross-attention. This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention; By synergistically integrating the recursive characteristics of bidirectional GRU with a multi-head cross-attention mechanism, a feature propagation network with both local temporal stability and global semantic awareness is constructed, which significantly enhances robustness to occluded regions and complex motion scenes. The arbitrary scale reconstruction module will enhance the temporal feature sequence. After further fusion via residual block concatenation, the data is then input into the Spatio-Temporal Adaptive Upsampling (STAU) module, based on the target scale factor. Generate the final high-resolution video frame sequence The STAU module includes a spatiotemporal filter branch and a scale filter branch: the former generates spatiotemporal filters for adaptive convolution based on implicit alignment features; the latter generates scale filters for adaptive convolution based on the current scale. This module dynamically generates upsampled convolution kernels by fusing scale information and spatiotemporal features, mapping low-resolution features to a high-resolution space, and achieving end-to-end arbitrary-scale video reconstruction. S4. Selection of Loss Function and Design of Evaluation Metric: The total loss function is a weighted sum of Charbonnier Loss and temporal gradient consistency loss, which simultaneously optimizes single-frame reconstruction quality and sequence temporal consistency. Charbonnier loss serves as the basic reconstruction loss, and by smoothing the L1 loss, it enhances the model's robustness to outliers while ensuring optimization stability. It measures the pixel-level difference between the reconstructed high-resolution frame and the true high-resolution frame, ensuring the reconstruction accuracy of a single frame. Its mathematical expression is: ; Where T represents the length (number of frames) of the video segment used during training. The total number of pixels in the image. and Let be the values ​​of the generated image and the real image at the i-th pixel, respectively, in frame t. It is the stability constant; Temporal gradient consistency loss is used to suppress inter-frame flicker and jitter. By constraining the inter-frame differences of the super-resolution sequence to remain consistent with the real sequence, it improves the temporal coherence and visual smoothness of the reconstructed video. Its mathematical expression is: ; in, and Let be the generated image and the real image of frame t, respectively. express Norm, used to calculate the sum of the absolute values ​​of inter-frame difference errors; Finally, a balance coefficient is introduced. To balance the ratio between the two loss functions, a total loss function is formed: ; The evaluation metrics used are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to verify the super-resolution performance and generalization ability of the model. Peak signal-to-noise ratio (PSNR) is based on the pixel error between the reconstructed image and the real image, and is used to objectively measure the reconstruction of the image. Its definition is based on mean squared error (MSE). ; ; Where L is the dynamic range of pixel values, which is 255 for an 8-bit image; Structural similarity comprehensively evaluates the similarity between images from three dimensions: brightness, contrast, and structure, and is used to measure the ability of reconstructed images to preserve structure. ; in, y and x are the mean values ​​of the image, respectively. For variance, For covariance, To maintain a stable constant.

[0025] All evaluation metrics are calculated in the luminance channel, consistent with mainstream video super-resolution research, to ensure the objectivity and comparability of experimental results; S5, Model Training and Fine-tuning: Train the video super-resolution reconstruction model at any scale. The preprocessed image data obtained in step S1 is processed in step S2. Bicubic downsampling is used to simulate the resolution degradation process in real-world scenes to obtain the corresponding low-resolution continuous video frames. These frames are then fed into the network model constructed in step S3 for training. The model is then fine-tuned to optimize the model parameters and further improve the super-resolution performance and generalization ability of the reconstruction model. During the training phase, the Vimeo-90K dataset was used as the training set as input to high-resolution video frames. For each training batch, consecutive frames were randomly sampled. A sequence of frames, where N is set to 3, meaning each training sample contains 7 consecutive frames. A target scale factor is also randomly generated. It covers integer scales (such as ×2, ×3, ×4), non-integer scales (such as ×1.5, ×2.25, ×2.7, ×3.3), and asymmetric scales (such as ×1.6 / ×3.05, ×3.5 / ×1.75, ×4 / ×1.4) to enhance the model's adaptability to diverse scale factors; The selected high-resolution video frame sequence is processed in step S2, and bicubic downsampling is performed according to the target scale factor s to generate the corresponding low-resolution video frame sequence. For non-integer or asymmetric scales, the boundaries of the high-resolution frames need to be appropriately cropped to ensure that the low-resolution frames generated after downsampling have integer sizes. For example, for a high-resolution frame with an original resolution of 576×704, if a ×2.5 downsampling is required, the high-resolution frame should first be cropped to 575×700 pixels, and then bicubic downsampling should be performed to generate a low-resolution frame with a size of 230×280 pixels. The generated low-resolution video frame sequence is input into the network model constructed in step S3. After step-by-step processing by modules such as feature extraction and alignment, temporal feature propagation and enhancement, and arbitrary scale reconstruction, the reconstructed high-resolution video frame sequence is output. The composite loss function designed in step S4 is used. Calculate the loss value between the reconstructed result and the true high-resolution frame, where the balance coefficient is... Set to 0.25; The model was trained using the Adam optimizer, with an initial learning rate set to... The batch size was set to 8; the training lasted for 300 epochs, with the learning rate halved at the 150th and 250th epochs to promote model convergence. During training, the network parameters were optimized using the backpropagation algorithm to minimize the total loss function, enabling the model to gradually learn the mapping relationship from low-resolution input to high-resolution output. After training, the model is fine-tuned using the UDM10 dataset 2 and Vid4 dataset 3 prepared in step S1. A smaller learning rate is used during the fine-tuning phase. The model was trained for 50 epochs to further adapt it to specific data distributions, optimize model parameters, suppress the accumulation of temporal errors in long sequence processing, and improve image reconstruction quality in occluded areas and complex motion scenes. Through the above training and fine-tuning strategies, an arbitrary-scale video super-resolution reconstruction model with strong robustness and high generalization ability was finally obtained. S6, Model Saving and Testing: After training and fine-tuning in step S5, the fine-tuned network parameters are fixed, and the final multi-frame video super-resolution reconstruction model is saved. The model is tested using the UDM10 and Vid4 datasets prepared in step S1. Real consecutive video frames are input into the trained end-to-end model to output reconstructed video super-resolution images at any scale. Testing is performed for symmetric and asymmetric scales separately, using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation metrics. Calculations are performed in the luminance channel to verify the model's super-resolution performance and generalization ability. Finally, the reconstruction performance of the model on multi-frame video super-resolution tasks at any scale is verified by calculating various evaluation metrics of the prediction results and ground truth labels. The performance metrics of this invention and existing technologies on the test sets of Dataset 2 and Dataset 3 are as follows: Figure 6 As shown, the peak signal-to-noise ratio and structural similarity of the present invention are significantly improved compared with the existing methods 1, 2 and 3;

[0026] Existing Method 1: This method employs an enhanced deformable convolutional network for video inpainting. Feature alignment and aggregation are achieved through a pyramid structure, a deformable alignment module, and a spatiotemporal attention fusion mechanism. Bicubic interpolation post-processing is then used to adjust the resolution to the target resolution. This method is tested using datasets 2 and 3 from step S1 of this invention.

[0027] Existing Method 2: Employs a bidirectional propagation scheme to maximize the extraction of temporal information, implements long-range dependency modeling through a cyclic architecture, and combines bicubic interpolation post-processing to adjust to the target resolution. This method was trained and tested using Dataset 2 and Dataset 3 from step S1 of this invention, respectively.

[0028] Existing Method 3: This method employs an iterative bidirectional architecture and scale-adaptive convolution to achieve arbitrary-scale video super-resolution. It dynamically adjusts feature extraction through full-dimensional scale attention convolution and combines spatiotemporal adaptive arbitrary-scale upsampling to achieve end-to-end reconstruction. The method is tested using datasets two and three from step S1 of this invention.

[0029] Under the same conditions, the feasibility and superiority of the proposed method were further verified by calculating the relevant indicators of the proposed method and existing methods on multi-frame super-resolution reconstruction tasks at arbitrary scales. The qualitative results of the existing technology and the method proposed in this invention are shown in the experimental figures. Figure 5 As shown in the figure, the quantitative results are presented in the experimental graph. Figure 6 As shown.

[0030] Furthermore, from Figure 5Qualitative analysis reveals that existing methods one and two, combined with post-processing, introduce significant blurring and artifacts during the interpolation process. Existing method three still falls short in detail recovery and temporal consistency under complex motion and occlusion scenarios. In contrast, the method proposed in this invention outperforms in contour edge, texture detail, and occlusion region recovery, achieving a visual effect closest to the real image. This is attributed to the unified spatiotemporal architecture of this invention, which avoids post-processing information loss. Combined with enhanced alignment modules and attention units, it ensures inter-frame stability while maintaining single-frame sharpness.

[0031] from Figure 6 Quantitative analysis shows that the proposed method achieves optimal performance across multiple symmetric and asymmetric scale tasks on the Vid4 and UDM10 datasets. For example, it improves PSNR by 0.37 dB at the ×1.5 scale and by 0.50 dB at the ×3.5 scale on the Vid4 dataset; and reaches 45.10 dB at the ×1.6 / ×3.05 scales on the UDM10 dataset. Comparative analysis indicates that existing methods one and two are limited by information loss in the two-stage process, while existing method three is insufficient in terms of alignment accuracy and error accumulation suppression. The proposed method effectively solves these problems, verifying its superiority in video super-resolution tasks at any scale.

[0032] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for arbitrary scale video super-resolution reconstruction based on error accumulation and occlusion, characterized in that, Includes the following steps: S1. Prepare the dataset: Prepare three video super-resolution benchmark datasets. Dataset 1 is used for network training and model fine-tuning, and Datasets 2 and 3 are used for model testing. S2, Construct training data pairs: Use bicubic downsampling to downsample high-resolution video frames at arbitrary scales to generate corresponding low-resolution video frame sequences, and construct input-output data pairs for network training; S3, Constructing a video super-resolution reconstruction network: The network model includes a feature extraction and alignment module, a temporal feature propagation and enhancement module, and an arbitrary scale reconstruction module; S4, Design of loss function and selection of evaluation index: Design loss function to minimize the difference between reconstruction result and real high-resolution image, so as to achieve efficient optimization of network parameters; The model performance and its generalization ability are evaluated based on the test data from datasets 2 and 3 in S1. Finally, multi-dimensional evaluation indicators are introduced to comprehensively measure the reconstruction accuracy and structural preservation ability of the model. S5, Model Training and Fine-tuning: After the training dataset preprocessed in S1 is processed in step S2, a low-resolution video frame sequence of the corresponding scale is generated, an input-output data pair is constructed, and then it is fed into the network model constructed in step S3 for training. During training, the model parameters are fine-tuned to further suppress the accumulation of temporal errors in long sequence processing and improve the image reconstruction quality in occluded areas and complex motion scenes. S6, Model Saving and Testing: After training and fine-tuning in step S5, solidify the network parameters and save the finally trained arbitrary-scale video super-resolution reconstruction model; use the real datasets 2 and 3 from step S1 to test the model. Multiple consecutive images of arbitrary scale can be directly input into the trained end-to-end model to obtain the reconstructed result image.

2. The method of claim 1, wherein the method is based on error accumulation and occlusion. In S1, dataset one is the Vimeo-90K dataset, dataset two is the UDM10 dataset, and dataset three is the Vid4 dataset.

3. The method of claim 1, wherein the method is based on error accumulation and occlusion. In step S2, training data pairs are constructed: high-resolution video frames are used as input, and downsampling is performed according to the target scale factor to generate corresponding low-resolution video frames; the scale factor can be set to integer, non-integer, or asymmetric values; for non-integer or asymmetric scales, the boundaries of the high-resolution frames are first cropped to ensure that low-resolution frames of integer size are generated after downsampling, and then low-resolution video frames are generated through bicubic downsampling.

4. The arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion according to claim 3, characterized in that: The double cubic down-sampling method is specifically: for a pixel point in a high-resolution image , a pixel value of a corresponding position in a low-resolution image is calculated through a double cubic interpolation function, and the double cubic interpolation function adopts the following kernel function: ; in, The distance between the point to be interpolated and its neighboring pixels is a parameter. The value is -0.5 or -1; the gray value of the target pixel in the low-resolution image is obtained by weighted summation of the gray values ​​of the pixels in the surrounding 4×4 neighborhood in the high-resolution image using the kernel function mentioned above.

5. The arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion according to claim 1, characterized in that: The feature extraction and alignment module in S3 uses shallow convolution with shared weights to extract the initial features of the input video sequence, and then guides the current frame features and the adjacent frame features input streams to the deformable alignment module for spatial alignment. The temporal feature propagation and enhancement module inputs the aligned features into the bidirectional recurrent attention module. By synergistically fusing the recursive characteristics of the bidirectional GRU and the multi-head cross attention mechanism, a feature propagation network with both local temporal stability and global semantic awareness is constructed. The arbitrary scale reconstruction module inputs the enhanced temporal features into the spatiotemporal adaptive arbitrary scale upsampling module after residual block cascading fusion, and generates the final high-resolution video frame sequence according to the target scale factor.

6. The arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion according to claim 5, characterized in that: The feature extraction and alignment module extracts initial features from the input video sequence using shallow convolutions with shared weights, and spatially aligns features from adjacent frames using a stream-guided deformable alignment module, given a low-resolution input video sequence. A total of 2N+1 consecutive images are generated. Initial features for each frame are first extracted using shallow convolutional layers with shared weights. Its expression is: ; in, This indicates a shallow convolution operation. This is the middle frame in a 2N+1 frame sequence. Then, the current frame features are... Features of adjacent frames The input stream guides the Flow-Guided Deformable Alignment (FGDA) module to perform spatial alignment. The alignment process can be represented as follows: ; in, For the features of the current frame, Features of adjacent frames A global mapping function for flow-guided deformable alignment modules; The flow-guided deformable alignment module consists of an offset prediction network, an optical flow estimation network, and deformable convolutions. The offset prediction network takes the channel concatenation of features from adjacent frames as input to predict the initial offset. Simultaneously, the SPyNet optical flow network is used to obtain the original image frames. and As input, predict the fundamental optical flow field. It provides global motion priors. The optical flow field and residual migration are fused point-by-point to obtain the final migration field. Then, adaptive alignment is performed using deformable convolution: ; in, To output the current pixel position on the feature map, The inherent first convolution kernel Each sampling offset, The weights of the k-th sampling point in deformable convolution are... The total offset field learned is in the th... Additional offset at each sampling point Original adjacent frame features; Aligned features from adjacent frames are concatenated with the current frame features along the channel dimension and then fed into a backend network that includes scale-adaptive convolution (Omni-Scale Convolution, OSConv). According to the target scale factor Perform feature enhancement and output the enhanced feature sequence. : ; in, These are the features of all frames after being aligned by the stream-guided deformable convolutional enhancement alignment module. It is the target scale factor. It is a backend network that includes scale-adaptive convolutions (OSConv); The temporal feature propagation and enhancement module will align the enhanced features. Input a bidirectional recurrent attention module to construct a feature propagation network that combines local temporal stability with global semantic awareness; The bidirectional cyclic attention module consists of three parts: a forward cyclic attention unit, a backward cyclic attention unit, and a bidirectional cross attention unit. The forward recurrent attention unit processes the sequence sequentially at the current time step in the forward time sequence. The input is the enhancement features of the current frame. Compared to the previous hidden state First, the current input and historical memory are fused through the gated recurrent unit (GRU). This process can be represented as follows: ; in, for The updated hidden state of the cell output at the current time step t. Enhance features for the current frame. This is the hidden state from the previous moment. For gated loop unit; Subsequently, by utilizing the cross-attention of multiple heads, For querying, use the historical hidden state sequence Perform a global context search for the key and value: ; in, The global context features are output by the multi-head cross-attention module. For multi-head cross-attention ensemble functions, Indicates from the first Frame to the The historical hidden state sequence of the frame, where N is the temporal window size; Finally, feature refinement is performed using residual connections and a multilayer perceptron, and the current hidden state is updated: ; in, The final refined feature output by the forward unit at time step t. This indicates that a normalization operation is performed after the residual connection. It is a multilayer perceptron. Indicates will The output hidden state is residually connected to the global context features output by multi-head cross-attention; Backward loop attention: The backward loop attention unit processes symmetric data, models the time sequence in reverse, and outputs backward features that contain future contextual information. ; The bidirectional cross-attention units perform forward-to-backward and backward-to-forward cross-attention operations respectively: ; ; in, This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention. This indicates that a normalization operation is performed after the residual connection. This indicates multiple points of cross-attention. The forward hidden features are the output of the forward recurrent attention unit. The backward hidden features are the output of the backward recurrent attention unit; The results from both approaches are then weighted and summed, and further fused using a multi-head self-attention layer and a multilayer perceptron (MLP) to output the final temporal features rich in global context. : ; in, The final output is a time-series feature rich in global context. This represents residual connection and layer normalization operations. It is a multilayer perceptron. This indicates multiple points of cross-attention. This is the output of the first bidirectional cross-attention. This is the output of the second bidirectional cross-attention; By synergistically integrating the recursive characteristics of bidirectional GRU with a multi-head cross-attention mechanism, a feature propagation network with both local temporal stability and global semantic awareness is constructed, which significantly enhances robustness to occluded regions and complex motion scenes. The arbitrary scale reconstruction module further fuses the enhanced temporal feature sequence through residual block concatenation, and then inputs it into the spatiotemporal adaptive arbitrary scale upsampling module to generate the final high-resolution video frame sequence according to the target scale factor. The spatiotemporal adaptive arbitrary-scale upsampling module includes a spatiotemporal filter branch and a scale filter branch. It dynamically generates upsampling convolution kernels by fusing scale information and spatiotemporal features, mapping low-resolution features to a high-resolution space. This process can be represented as: ; in, This represents the target pixel location on the high-resolution feature map. for The corresponding projection position on the low-resolution feature map For local domain windows; The spatiotemporal adaptive convolution weights are generated by the spatiotemporal filter branch. The scale-adaptive convolution weights (dependent on the target scale factor s) are generated by the scale filter branch. For learnable offsets, The feature value at the sampling point; The spatiotemporal filter branch generates spatiotemporal adaptive convolution weights using the enhanced temporal feature sequence as input. The scaling filter branch generates scale-adaptive modulation weights using the target scaling factor s as input. Finally, the two are multiplied point by point to form a joint upsampling convolution kernel, which outputs a high-resolution video frame sequence.

7. The arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion according to claim 1, characterized in that: In the design of the evaluation metrics, the S4 loss function uses a weighted sum of Chabonie loss and temporal gradient consistency loss as the total loss function; the evaluation metrics used are peak signal-to-noise ratio and structural similarity. The Charbonnier Loss is used to measure the pixel-level difference between the reconstructed high-resolution frame and the real high-resolution frame, ensuring the reconstruction accuracy of a single frame image. The temporal gradient consistency loss is used to constrain the inter-frame differences of the super-resolution sequence to remain consistent with the real sequence, thereby improving the temporal coherence and visual smoothness of the reconstructed video. The evaluation metrics used are peak signal-to-noise ratio and structural similarity to verify the super-resolution performance and generalization ability of the model. All metrics are calculated in the luminance channel.

8. The arbitrary-scale video super-resolution reconstruction method based on error accumulation and occlusion according to claim 7, characterized in that: The Chabonnier loss is defined as follows: ; Where T is the length of the video segment used during training. The total number of pixels in the image. and Let be the values ​​of the generated image and the real image at the i-th pixel, respectively, in frame t. It is the stability constant; The temporal gradient consistency loss is defined as follows: ; in, and Let be the generated image and the real image of frame t, respectively. express Norm, used to calculate the sum of the absolute values ​​of inter-frame difference errors; Finally, a balance coefficient is introduced. To balance the ratio between the two loss functions, a total loss function is formed: ; The peak signal-to-noise ratio (PSNR) is based on the pixel error between the reconstructed image and the real image, and is used to objectively measure the reconstruction of the image. Its definition is based on the mean square error (MSE). ; ; Where N represents the total number of pixels in the image, and i is the pixel index. Let be the pixel value at the i-th pixel in the super-resolution reconstructed image of frame t. Let L be the pixel value of the real high-resolution image in frame t at the i-th pixel, and L be the dynamic range of the pixel value, which is 255 for an 8-bit image. Structural similarity comprehensively evaluates the similarity between images from three dimensions: brightness, contrast, and structure, and is used to measure the ability of reconstructed images to preserve structure. ; in, y and x are the mean values ​​of the image, respectively. For variance, For covariance, To maintain a stable constant.

Images