Video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction

Abstract

The invention belongs to the technical field of video restoration, and particularly relates to a video super-resolution reconstruction method based on self-adaptive exit and frequency domain Gaussian reconstruction, which comprises the following steps: establishing an MGF-VSR network framework, introducing a motion guide mask, a basic reconstruction branch, a Gaussian reconstruction branch, time sequence consistency fusion and a self-adaptive mask updating mechanism, and reconstructing the video super-resolution based on the self-adaptive exit and the frequency domain Gaussian reconstruction. Differential modeling of a static region and a dynamic region in a video is realized, so that the reconstruction quality and the time sequence stability are remarkably improved, and the method can be compatible and expanded with an existing video super-division, video deblurring or video compression reconstruction framework, and has wide engineering application potential.

Description

Technical Field

[0001] This invention belongs to the field of video restoration technology, specifically relating to a video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction. Background Technology

[0002] Early video super-resolution (VSR) methods were primarily based on signal processing and statistical modeling principles. Liu and Sun et al. proposed a Bayesian inference-based video super-resolution algorithm that reconstructs high-resolution sequences by explicitly estimating the inter-frame motion field. These algorithms rely on strong assumptions about camera motion, illumination variations, and noise distribution. While they achieve good restoration results in simulated environments, they exhibit poor robustness in real-world, complex scenes.

[0003] With the development of deep learning technology, researchers have gradually shifted from explicit modeling to end-to-end feature learning. VESPCN adopts a joint architecture of convolution and spatial transformation, which can automatically capture the spatiotemporal correlation between video frames through a learning module. DRVSR introduces a dynamic reference frame selection strategy based on an attention mechanism, which adaptively selects the best neighboring frames for reconstruction based on the features of the current frame. FRVSR proposes a feature pyramid structure to efficiently capture multi-scale spatiotemporal information, but its network complexity is high. STTN constructs a spatio-temporal attention network (Spatio-Temporal Transformer Network), which significantly improves the ability to model cross-frame features, but the computational cost is high. SOFVSR designs a self-supervised optical flow-assisted loss to learn temporal consistency under unlabeled conditions. TOFlow proposes a temporal optical flow fusion module, which significantly improves motion alignment. MEMC-Net combines motion estimation and compensation to improve the accuracy of cross-frame feature fusion.

[0004] In recent years, researchers have begun to explore joint modeling of frequency domain information and spatial domain features. Zhu et al. proposed a frequency domain deblurring framework based on Fourier transform, revealing the characteristics of blurred videos in the frequency domain energy spectrum distribution. Mao et al. found that frequency filtering can reflect the direction and intensity of blur, and embedded Fourier transform modules in the residual block to achieve dual-domain learning. Yu et al. introduced a Gaussian-weighted cross-frame nonlocal attention mechanism to achieve inter-frame information fusion without explicit alignment.

[0005] The ETDM framework proposed by Isobe et al. divides low-difference and high-difference regions through a difference modeling module and processes them separately using branches with different receptive fields. It jointly extracts the optimal features from past and future frames to achieve more stable cross-frame compensation.

[0006] Xu et al.'s VideoGigaGAN introduces a flow-guided feature propagation and a high-frequency feature carrier mechanism, balancing high-frequency detail reconstruction with temporal consistency. This work demonstrates that joint modeling in the frequency and spatial domains has significant advantages in improving reconstructed detail and enhancing model generalization.

[0007] Despite significant progress made by existing methods in video super-resolution reconstruction, the following technical bottlenecks still exist:

[0008] Limitations of motion modeling: Under complex motion (such as non-rigid deformation, occlusion, dynamic background), traditional optical flow and deformable convolution are difficult to accurately capture high-order motion features, resulting in unstable feature alignment and blurred details.

[0009] Insufficient temporal consistency: Most models only perform simple frame fusion at the feature level, lacking explicit modeling of time dependencies or adaptive exit mechanisms, which easily leads to cross-frame artifacts and temporal flicker.

[0010] High computational and storage overhead: Multi-frame loop propagation or Transformer structure improves spatiotemporal modeling capabilities, but introduces high computational complexity and memory requirements, which is not conducive to real-time applications.

[0011] Poor adaptability to high-resolution videos: Existing training data is mainly concentrated in low-resolution videos such as Vimeo90K and Vid4, and the transfer performance of the model in high-resolution scenes such as 2K, 4K, and 8K is significantly reduced.

[0012] Insufficient ability to recover compressed videos: Most methods are only trained on uncompressed sources, and are prone to block artifacts and artifact spread when faced with real compressed videos. Summary of the Invention

[0013] To address the aforementioned technical problems, this invention provides a video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction, comprising:

[0014] S1. Establish an MGF-VSR network framework, which includes: an optical flow estimation and motion region segmentation module, a mask update module, a frequency domain Gaussian reconstruction module, a backbone reconstruction network, and a temporal consistency constraint and gain-aware fusion module.

[0015] S2. Input the low-resolution video frame sequence into the optical flow estimation and motion region segmentation module to extract basic features from the low-resolution video sequence, estimate inter-frame optical flow, and generate motion masks.

[0016] S3. Basic image reconstruction is performed based on basic features and motion masks through a backbone reconstruction network;

[0017] S4. Generate high-frequency detail maps based on fundamental features using the frequency domain Gaussian reconstruction module;

[0018] S5. The base image and high-frequency detail image are dynamically fused through the temporal consistency constraint and gain-aware fusion module, and inter-frame temporal correction is performed based on inter-frame optical flow.

[0019] S6. After each frame of video reconstruction, the mask update module dynamically updates the motion mask for the next iteration based on the current reconstruction quality.

[0020] S7. Repeat steps S2-S6 until every frame of the low-resolution video has been reconstructed, resulting in the final reconstructed video.

[0021] Compared with the prior art, the present invention has the following significant advantages and positive effects:

[0022] Motion Adaptive Exit Mechanism: Intra-block early exit is implemented through the Mask Update Module (MUM), which enables the network to automatically skip redundant calculations in fully converged regions while ensuring reconstruction quality. In actual inference, this can reduce the amount of computation by about 30%–45%.

[0023] Frequency Domain Gaussian Detail Reconstruction: The Gaussian reconstruction branch can reconstruct smoother and more continuous high-frequency details under low complexity conditions, overcoming the texture fragmentation and edge artifacts caused by traditional convolution upsampling.

[0024] Enhanced temporal consistency: Optical flow-guided temporal consistency constraints effectively suppress cross-frame flicker, ensuring that the output video sequence is coherent and natural in the temporal dimension.

[0025] Gain-aware fusion optimization: Through a gain-aware gating mechanism, spatial and frequency domain information are adaptively fused to improve the recovery of complex motion regions, occluded edges and fine textures.

[0026] Lightweight, efficient, and adaptable to high resolution: By adopting a modular design and parallel branching structure, the number of model parameters and inference latency are significantly lower than existing Transformer-type methods.

[0027] Versatility and scalability: The masking mechanism and Gaussian reconstruction module of this invention are compatible with and can be extended to existing video super-resolution, video deblurring, or video compression and reconstruction frameworks, and have broad engineering application potential. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention.

[0029] Figure 2This is a network structure diagram of the FEMM (Flow Estimation & Motion Mapping) module of the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention.

[0030] Figure 3 This is a structural diagram of the BRN (Backbone Reconstruction Network) and its core submodule IIM (Interaction and Integration Module) of the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention.

[0031] Figure 4 This is a structural diagram of the GRB (Gaussian Reconstruction Branch) module of the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention;

[0032] Figure 5 This is a structural diagram of the GAF-TCC (Gain-Aware Fusion & Temporal Consistency Constraint) module of the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention.

[0033] Figure 6 This is a visual comparison diagram between existing methods and the video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] A video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction, such as Figure 1 As shown, it includes:

[0036] S1. Establish an MGF-VSR network framework, which includes: an optical flow estimation and motion region segmentation module, a mask update module, a frequency domain Gaussian reconstruction module, a backbone reconstruction network, and a temporal consistency constraint and gain-aware fusion module.

[0037] S2. Input the low-resolution video frame sequence into the optical flow estimation and motion region segmentation module to extract basic features from the low-resolution video sequence, estimate inter-frame optical flow, and generate motion masks.

[0038] S3. Basic image reconstruction is performed based on basic features and motion masks through a backbone reconstruction network;

[0039] S4. Generate high-frequency detail maps based on fundamental features using the frequency domain Gaussian reconstruction module;

[0040] S5. The base image and high-frequency detail image are dynamically fused through the temporal consistency constraint and gain-aware fusion module, and inter-frame temporal correction is performed based on inter-frame optical flow.

[0041] S6. After each frame of video reconstruction, the mask update module dynamically updates the motion mask for the next iteration based on the current reconstruction quality.

[0042] S7. Repeat steps S2-S6 until every frame of the low-resolution video has been reconstructed, resulting in the final reconstructed video.

[0043] Optical Flow Guided Motion Region Segmentation (FEMM): In video as seen by the human eye, some areas remain almost stationary (background, walls), while others change dramatically (arm swings, moving objects). The reconstruction difficulty varies depending on the region.

[0044] Static areas: are relatively easy to rebuild;

[0045] Motion areas require higher computational power to avoid ghosting and blurring;

[0046] Figure 2 The network structure of the FEMM (Flow Estimation & Motion Mapping) module is demonstrated. The input is a low-resolution frame sequence (B, T, 3, H, W). This module first passes through two... The +ReLU layer extracts shallow features, and then uses a SpyNet-based pyramid optical flow estimator to extract optical flow between consecutive frames. The module then calculates motion amplitude and divides motion regions based on the amplitude, generating corresponding Motion Area Maps and binary Motion Mask Maps. This module automatically identifies motion regions in a video and evaluates the degree of motion. The input is a low-resolution frame sequence (B, T, 3, H, W). This module first processes two... The module uses a ReLU layer to extract shallow features, and then extracts the optical flow between consecutive frames using a SpyNet-based pyramid optical flow estimator. The module then calculates the motion amplitude and divides the motion region based on the amplitude, generating corresponding Motion Area Maps and binary Motion Mask Maps. This module processes input low-resolution video frame sequences. Three processes were performed:

[0047] Basic Feature Extraction: Two 3×3 convolutional layers are used to perform a "shallow understanding" of the input image, similar to preprocessing the image to make it easier to determine motion later. Primary spatial features are obtained through these two 3×3 convolutional layers and used as a unified starting point for the subsequent two branches.

[0048]

[0049] in, Indicates basic features, This represents a low-resolution video frame sequence. express Convolutional layer This indicates a modified linear unit activation function. Convolutional layers are used to extract basic visual features such as edges and textures, and the output is... This then serves as the input for the subsequent BRN and GRB branches.

[0050] Optical flow estimation: To determine motion, the positional difference of the same pixel in two consecutive frames must be compared. The SpyNet optical flow network is used to calculate the direction and magnitude of pixel movement between adjacent frames.

[0051]

[0052] in, This represents the optical flow field from time t to t+1. , These represent the low-resolution input frames at times t and t+1, respectively. This represents a pre-trained SpyNet optical flow estimation network;

[0053] Wherein: if the pixel is stationary → optical flow ≈ 0, if the pixel moves violently → optical flow value is large.

[0054] Motion region segmentation: Based on the optical flow amplitude, the spatial position is divided into 5 levels of motion intensity and converted into "strong motion / medium motion / weak motion / static" region masks.

[0055] First, calculate the length (i.e., motion velocity) of the optical flow vector to obtain the motion intensity value (grayscale image form) for each pixel:

[0056]

[0057] in, This represents the optical flow vector at pixel (x,y) from frame t to frame t+1, containing both horizontal and vertical components. This represents the 2-norm of a vector.

[0058] Then, the motion intensity is divided into five levels: stationary area → slight movement → medium-speed movement → fast movement → vigorous movement. A binary motion mask is then generated based on the threshold values.

[0059]

[0060] in, Represents pixels The initial motion mask, Represents pixels The intensity of motion at time t, This indicates the preset exercise intensity threshold.

[0061] The mask is used to guide the dynamic computation path of the subsequent backbone network and Gaussian branches, enabling selective processing at the spatial block level.

[0062] This module outputs the following features: , optical flow Initial motion mask .

[0063] Backbone Reconstruction Network (BRN): The BRN is responsible for reconstructing the "overall picture," including:

[0064] Larger areas of texture; shape and structural outlines; smoothing information such as brightness and shadows;

[0065] Figure 3 This demonstrates the structure of the BRN (Backbone Reconstruction Network) and its core sub-module, the IIM (Interaction and Integration Module). The left side shows the BRN backbone reconstruction network, employing a U-shaped symmetrical structure composed of three encoder layers (Encoder Blocks 1–3) and three decoder layers (Decoder Blocks 3–1), with features transferred via skip connections. Each layer contains multiple IIM modules (IIM×4, ×8, ×12) to achieve mask-guided implicit temporal feature fusion. Finally, after… Output basic reconstruction results The right side shows the structural details of the IIM: input features. First through LayerNorm, , Spatial information is extracted and then gated and fused with a motion mask from FEMM during the depthwise convolution stage. This is followed by two layers of... Feedforward channel integration (FFN) and output via residual connection In the picture ⊕ represents depthwise separable convolution; ⊕ represents residual addition. BRN adopts an encoder-decoder structure, similar to the U-Net network, and uses a symmetrical encoder-decoder structure (EB1–3, DB1–3). It continuously downsamples in the encoder stage to extract the global structure from the local texture, and inserts IIM (Interaction Integration Module) between each encoder and decoder stage for motion-guided attention fusion.

[0066] In each network level, IIM performs the following operations:

[0067]

[0068] in, This represents element-wise multiplication. Indicates motion mask The downsampled version in the feature space during the t-th iteration. This indicates that the backbone reconstruction network is in the [number]th [year]. The input feature map of the layer can be regarded as the basic feature. The intermediate representation after the first few convolutional and downsampling layers; and This represents the output feature map after processing by the IIM module. Indicates the IIM module. Represents the Gaussian error linear unit activation function. This indicates a feature normalization operation. This represents a 3D convolution operator performed on a temporally stacked feature sequence, used to simultaneously model spatial and temporal neighborhood information.

[0069] In the decoder stage, features are recovered from low dimension back to high resolution. Finally, the BRN outputs the basic reconstruction result:

[0070]

[0071] in, Represents the basic image. Indicates basic features, Indicates the initial motion mask. This indicates that the backbone network is being rebuilt.

[0072] Gaussian Reconstruction Branch (GRB): The role of GRB is to generate high-frequency details: fine textures; edge sharpness; complex patterns;

[0073] BRN details that are difficult to recover; Figure 4 The structure of the GRB (Gaussian Reconstruction Branch) module is shown. This module receives features from the FEMM. Gaussian parametric modeling and pixel-level high-frequency reconstruction are achieved through three-layer convolution and a vectorized RendererFast renderer. Specifically, the module first undergoes... and Local features are extracted, and then RendererFast implements 2D Gaussian blending rendering using a fully vectorized approach of unfold + einsum. Finally, through... Map the K Gaussian components to a three-channel color output. GRB employs a learnable Gaussian mixture modeling structure to compensate for high-frequency details and non-linear textures that are difficult for BRN to recover. This module receives features from FEMM. Gaussian parametric modeling and pixel-level high-frequency reconstruction are achieved through three-layer convolution and a vectorized RendererFast renderer. Specifically, the module first undergoes... and Local features are extracted, and then RendererFast implements 2D Gaussian blending rendering using a fully vectorized approach of unfold + einsum. Finally, through... Map the K Gaussian components to a three-channel color output. .

[0074] The first step is to generate Gaussian parameters, mapping the features to K sets of two-dimensional Gaussian kernel parameters. Input features A Gaussian parameter set is generated through three 1×1 convolutional layers:

[0075]

[0076] in, Indicates basic features, This represents a 1×1 convolutional layer. Indicates the number of Gaussian kernels. These represent the location, variance, and intensity parameters, respectively.

[0077] Image details will be formed by linearly superimposing these Gaussian kernels.

[0078] Then, a vectorized Gaussian Renderer (RendererFast) is used, and the two-dimensional Gaussian kernel is calculated according to the formula:

[0079]

[0080] in, and These represent the center positions of the k-th Gaussian kernel in the horizontal and vertical directions, respectively. This indicates that the k-th Gaussian kernel is located at pixel position. The response value generated at that location;

[0081] RendererFast does not loop pixel by pixel. Instead, it uses: unfold to unfold the entire image into matrix blocks; and einsum to perform batch Gaussian operations. Its advantages are that it is faster, more stable, and can handle larger resolutions.

[0082] Finally, we obtain the high-frequency detail map, which is a "detail enhancement map" that reveals more texture:

[0083]

[0084] in, Represents high-frequency detail images. This represents the response value of the k-th Gaussian kernel.

[0085] This branch achieves pixel-level parallel computation through a fully vectorized renderer (RendererFast), avoiding explicit convolution and loop operations.

[0086] Figure 5 The structure of the GAF-TCC (Gain-Aware Fusion & Temporal Consistency Constraint) module is shown. This module receives data from the BRN. With GRB As input, the module adaptively fuses the two reconstruction results and performs inter-frame temporal correction. It learns dynamic fusion weights based on the texture complexity and motion intensity of the local region.

[0087] First, the two results are concatenated, and then a dynamic weight map W is generated through convolution and activation.

[0088] W

[0089] Among them, Represents a dynamic weighted graph. This represents a 1×1 convolutional layer. express Convolutional layer This indicates a modified linear unit activation function. express, Represents high-frequency detail images. Represents the basic image. This represents the Sigmoid activation function;

[0090] The final fusion form is as follows:

[0091]

[0092] in, This indicates the fused reconstruction result of the previous frame. This represents an operator that performs distorted resampling of an image based on optical flow, used to align the result of the previous frame to the coordinate system of the current frame. This represents the aligned and merged image from the previous frame. Let represent the optical flow field from frame t−1 to frame t;

[0093] Next, we use the optical flow field of the FEMM module to align the results from the previous frame:

[0094]

[0095] It also forces the current frame to be similar to the aligned previous frame to avoid flickering and texture jitter:

[0096]

[0097] The Mask Update Module (MUM) updates the mask state in each inference round based on the improvement in PSNR and LPIPS, achieving block-level adaptive exit. When the performance gain of a spatial block falls below a threshold, that block stops deep computation in subsequent iterations.

[0098] The MUM module is used to answer two questions: Which regions are "well recovered" → computation can be reduced; which regions are "still not good enough" → further processing is needed in subsequent iterations.

[0099] It automatically adjusts the motion mask for the next round based on the reconstruction quality, achieving block-level adaptive exit.

[0100] MUM refers to the actual quality gain during training, based on: PSNR improvement. LPIPS reduction ;

[0101] "PSNR Improvement ΔPSNR" indicates the magnitude of the improvement in peak signal-to-noise ratio (PSNR) for this spatial block in the current iteration compared to the previous iteration; "LPIPS Decrease ΔLPIPS" indicates the magnitude of the decrease in perceptual similarity index (LPIPS) for this spatial block compared to the previous iteration (lower LPIPS indicates better visual quality). These two quantities measure the improvement in pixel fidelity and perceptual quality of image reconstruction, respectively, and are the basis for evaluating whether it is still valuable to continue deep inference in this region.

[0102] Update the motion mask used in the next iteration:

[0103]

[0104] If the gain of a certain region is lower than the threshold, it is considered to have "converged", and deep calculations can be skipped in subsequent inference, realizing adaptive block-level early exit.

[0105] For each block Define the gain scoring function:

[0106]

[0107] when When the mask value is set to 0, it indicates an early exit; otherwise, it is retained as 1.

[0108] The mask matrix is ​​continuously updated throughout the video inference process and stored as mask.npy, enabling continuous inter-frame adaptation.

[0109] Specifically, for each space block This invention calculates a comprehensive gain score based on the quality improvement obtained in the current iteration relative to the previous iteration. .in, , and These represent the improvements in pixel fidelity, perceived quality, and motion consistency achieved by the block, respectively. , , These are the corresponding weighting coefficients, used to balance the importance of the three indicators. If a region's score... Less than the preset exercise intensity threshold If the mask value for that region is zero, it means that the quality gain from continuing deep network computation is minimal, and it can be considered to have "converged." Therefore, in the next iteration, the mask value for that region is set to 0, preventing it from passing through the deep network again and thus exiting the computation early. Conversely, if... ≥ This indicates that there is still room for improvement in this area, so its mask value is kept at 1, allowing it to continue undergoing deeper reconstruction. In this way, a block-level adaptive exit strategy based on quality gain is achieved.

[0110] In this embodiment, an experimental verification of a video super-resolution reconstruction method based on adaptive exit and frequency domain Gaussian reconstruction is conducted.

[0111] Experimental setup:

[0112] Datasets and Evaluation Metrics: This invention was validated on the Vimeo90K, REDS, and Vid4 datasets. During training, subsets of Vimeo90K and GoPro training datasets (containing sequences with diverse motion and fuzzy distributions) were used. During testing, model performance was evaluated on Vimeo-Test, REDS4, and GoPro-Test, respectively. Evaluation metrics used included Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Params.

[0113] Training settings: The optimizer uses AdamW, and the parameters are set as follows: =0.9, =0.99, initial learning rate 2× And a cosine annealing strategy was used to reduce it to 1× .

[0114] The training clipping block size is 256×256, the batch size is 8, the window length is 7, and data augmentation includes random horizontal flipping and rotation.

[0115] The total loss function is defined as:

[0116]

[0117] in, : Basic reconstruction L1 loss; Time consistency loss; Frequency domain Charbonnier loss; : Control prior losses.

[0118] Experimental Results: To demonstrate the superiority of the proposed method, the network proposed in this invention was compared with existing, highly representative image restoration methods. The comparison literature and results are shown in Tables 1 and 2, respectively.

[0119] Table 1 Comparison Methods

[0120] Method name abbreviation Comparative Literature VSRM (ICCV'25) Tran DP, Hung DD, Kim D. VSRM: A Robust Mamba-Based Framework for Video Super-Resolution[J]. arXiv preprint arXiv:2506.22762, 2025. IART (CVPR'24) Xu K, Yu Z, Wang X, et al. Enhancing video super-resolution via implicitresampling-based alignment[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2024: 2546-2555. MIA-VSR (CVPR'24) Zhou X, Zhang L, Zhao X, et al. Video super-resolution transformer with maskedinter&intra-frame attention[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2024: 25399-25408. MFPI (ICCV'23) Li F, Zhang L, Liu Z, et al. Multi-frequency representation enhancement with privilege information for video super-resolution[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision. 2023: 12814-12825. MambaVSR (arXiv'25) He L, Liu M, Tang Q, et al. MambaVSR: Content-Aware Scanning State Space Model forVideo Super-Resolution[J]. arXiv preprint arXiv:2506.11768, 2025. PSRT(NeurIPS'22) Shi S, Gu J, Xie L, et al. Rethinking alignment in video super-resolutiontransformers[J]. Advances in Neural Information Processing Systems, 2022, 35:36081-36093. DFVSR(IJCAI'23) Dong S, Lu F, Wu Z, et al. DFVSR: Directional Frequency Video Super-Resolution viaAsymmetric and Enhancement Alignment Network[C] / / IJCAI. 2023: 681-689. RVRT(NeurIPS'22) Liang J, Fan Y, Xiang X, et al. Recurrent video restoration transformer withguided deformable attention[J]. Advances in Neural Information Processing Systems,2022, 35: 378-393. VRT(TIP'24) Liang J, Cao J, Fan Y, et al. Vrt: A video restoration transformer[J]. IEEETransactions on Image Processing, 2024, 33: 2171-2182. TTVSR(CVPR'22) Liu C, Yang H, Fu J, et al. Learning trajectory-aware transformer for video super-resolution[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition. 2022: 5687-5696. BasicVSR++(CVPR'22) Chan K C K, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution withenhanced propagation and alignment[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2022: 5972-5981. ETDM(CVPR'22) Isobe T, Jia X, Tao X, et al. Look back and forth: Video super-resolution withexplicit temporal difference modeling[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition. 2022: 17411-17420. IconVSR(CVPR'21) Chan K C K, Wang X, Yu K, et al. Basicvsr: The search for essential components invideo super-resolution and beyond[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2021: 4947-4956. BasicVSR(CVPR'21) Chan K C K, Wang X, Yu K, et al. Basicvsr: The search for essential components in video super-resolution and beyond[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 4947-4956.

[0121] Table 2 Comparison Results

[0122] Method Frames REDS / Vimeo Params(M) REDS4 PSNR REDS4 SSIM Vimeo-90K-T PSNR Vimeo-90K-T SSIM Vid4 PSNR Vid4 SSIM VSRM 16 / 7 17.1 33.11 0.9162 38.33 0.9543 28.44 0.8552 IART 16 / 7 13.4 32.90 0.9138 38.14 0.9528 28.26 0.8517 MIA-VSR 16 / 14 16.5 32.78 0.9220 38.22 0.9532 28.20 0.8507 MFPI - / - 7.3 32.81 0.9106 38.28 0.9534 28.11 0.8481 MambaVSR 16 / 14 14.1 32.75 0.9110 38.33 0.9539 28.20 0.8514 PSRT-recurrent 16 / 14 13.4 32.72 0.9106 38.27 0.9536 28.07 0.8485 DFVSR - 7.1 32.76 0.9081 38.25 0.9556 27.92 0.8427 RVRT 30 / 14 10.8 32.75 0.9113 38.15 0.9527 27.99 0.8426 VRT 16 / 7 35.6 32.19 0.9006 38.20 0.9530 27.93 0.8425 TTVSR 50 / - 6.8 32.12 0.9021 37.92 0.9526 28.40 0.8643 BasicVSR++ 30 / 14 7.3 32.39 0.9069 37.79 0.9500 27.79 0.8400 ETDM - 8.4 32.15 0.9024 - - 28.81 0.8725 IconVSR 15 / 14 8.7 31.67 0.8948 37.47 0.9476 27.39 0.8279 BasicVSR 15 / 14 6.3 31.42 0.8909 37.18 0.9450 27.24 0.8251 MGF-VSR (Ours) 16 / 7 13.2 33.16 0.9177 38.36 0.9516 28.49 0.8693

[0123] As shown in Table 2 and Figure 6 As shown, this invention provides a systematic quantitative comparison of representative video super-resolution restoration methods in recent years from aspects such as average PSNR, average SSIM, and the number of model parameters.

[0124] Experiments were conducted on publicly available benchmark datasets such as REDS4, Vimeo-90K-T, and Vid4, with all models being compared fairly under the same training and inference settings.

[0125] As can be seen from Table 2, when the model is trained on the Vimeo-90K and REDS mixed training set, the MGF-VSR (Motion-aware Gaussian Fusion Video Super-Resolution) proposed in this invention achieves the best performance with lower computational cost in most cases.

[0126] Specifically:

[0127] On the REDS4 dataset, MGF-VSR achieved an average PSNR of 33.16 dB and an average SSIM of 0.9177, which is 0.41 dB higher than MambaVSR (32.75 dB) and 0.77 dB higher than BasicVSR++ (32.39 dB), and achieved higher temporal consistency and reconstruction accuracy under similar parameter scales.

[0128] On the Vimeo-90K-T dataset, MGF-VSR achieved 38.36 dB / 0.9516 SSIM, comparable to VSRM (38.33 dB / 0.9543) and MambaVSR (38.33 dB / 0.9539), but with only 13.2 M parameters, about 22.8% less than VSRM and about 6% less than MambaVSR, making it the best in terms of balancing lightweight design and performance.

[0129] On the Vid4 dataset, MGF-VSR achieves 28.49 dB / 0.8693 SSIM, which is 0.39 dB / 0.0186 SSIM higher than MIA-VSR (28.10 dB / 0.8507), further demonstrating the stability and universality of the method in real video reconstruction tasks.

[0130] Overall, MGF-VSR outperforms existing mainstream Transformer and Mamba series models in both PSNR and SSIM metrics, achieving lightweight, high-fidelity, and strong temporal consistency performance with only 13.2 M parameters.

[0131] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for video super-resolution reconstruction based on adaptive exit and frequency domain Gaussian reconstruction, characterized in that, Comprise: S1, establish MGF-VSR network framework, the MGF-VSR network framework, comprising: optical flow estimation and motion region division module, mask update module, frequency domain Gaussian reconstruction module, backbone reconstruction network, timing consistency constraint and gain perception fusion module; S2, the low-resolution video frame sequence is input into the optical flow estimation and motion region division module, the basic feature is extracted from the low-resolution video sequence, the interframe optical flow is estimated and the motion mask is generated; S3, based on the basic feature and the motion mask, the basic image is reconstructed through the backbone reconstruction network; S4, the high-frequency detail map is generated based on the basic feature through the frequency domain Gaussian reconstruction module; S5, the basic image and the high-frequency detail map are dynamically fused through the timing consistency constraint and gain perception fusion module, and the interframe timing correction is carried out based on the interframe optical flow; S6, after each frame of video is reconstructed, the mask update module dynamically updates the motion mask of the next round of iteration according to the current reconstruction quality; S7, repeat steps S2-S6 until each frame of the low-resolution video is reconstructed, and the final reconstructed video is obtained.

2. The method of claim 1, wherein, The low-resolution video frame sequence is input into the optical flow estimation and motion region division module, the basic feature is extracted from the low-resolution video sequence, the interframe optical flow is estimated and the motion mask is generated, Comprise: Basic feature extraction: wherein, denotes a base feature, denotes a sequence of low resolution video frames, denotes convolutional layer, denotes a rectified linear unit activation function; Optical flow estimation: wherein, denotes the optical flow field from time t to t+1, , denote low resolution input frames at time t and t+1, respectively, denotes a pre-trained SpyNet optical flow estimation network; Motion mask generation: wherein, represents a pixel of an initial motion mask, represents a pixel of a motion intensity at time t, represents a preset motion intensity threshold.

3. The method of claim 1, wherein, Based on the basic feature and the motion mask, the basic video is reconstructed through the backbone reconstruction network, comprising: The backbone reconstruction network is a symmetrical Encoder-Decoder structure, comprising: three encoding blocks and three decoding blocks, and an IIM module is inserted between each level of encoder and decoder; The base features are input together with the initial motion mask to the first layer encoder for feature encoding, and the encoder obtains the features at the first layer, denoted as , and performs spatial down-sampling through convolution with a step size of 2; meanwhile, the mask is obtained at the corresponding scale using the same down-sampling manner ; the IIM module inserted between each level of the encoder and the decoder takes the features as input, applies different attention weights to the motion region and the static region, and outputs the enhanced features ; At the decoder stage, the low-resolution features output by the encoder are progressively upsampled and fused with the same-scale encoded features and the IIM output through skip connections, allowing high-level global structure information and low-level detail texture to jointly participate in reconstruction; after multi-level encoding-decoding and IIM fusion, the base reconstructed image is output at the last decoder .

4. The method of claim 3, wherein, Motion-guided attention fusion is carried out through the IIM module in the downsampling process, comprising: in, This represents element-wise multiplication. Indicates motion mask The downsampled version in the feature space during the t-th iteration. This indicates that the backbone reconstruction network is in the [number]th [year]. The input feature map of the layer, This represents the output feature map after processing by the IIM module. Indicates the IIM module. Represents the Gaussian error linear unit activation function. This indicates a feature normalization operation. This represents a three-dimensional convolution operator performed on a temporally stacked feature sequence.

5. The method of claim 3, wherein the method further comprises: The features are recovered from low dimension to high resolution through the decoder and the IIM module layer by layer, and the basic reconstruction result is obtained, comprising: wherein, denotes a base image, denotes a base feature, denotes an initial motion mask, denotes a backbone reconstruction network.

6. The method of claim 1, wherein, Based on the basic feature, the high-frequency detail map is generated through the frequency domain Gaussian reconstruction module, Comprise: input features A set of Gaussian parameters is generated by three 1x1 convolutional layers: where, represents the base feature, represents the 1x1 convolutional layer, represents the number of Gaussian kernels, and, respectively, represent the position, variance, and intensity parameters; Vectorize the Gaussian parameter set: where, and denote the center position of the k-th Gaussian kernel in the horizontal and vertical directions, respectively, denotes the response value generated by the k-th Gaussian kernel at pixel position . generate a high frequency detail map from the set of Gaussian parameters after the vector: wherein, denotes the high frequency detail map, denotes the response value of the k-th Gaussian kernel.

7. The method of claim 6, wherein, The basic image and the high-frequency detail map are dynamically fused through the timing consistency constraint and gain perception fusion module, and the interframe timing correction is carried out based on the interframe optical flow, comprising: The base image and the high-frequency detail image are spliced, and a dynamic weight image is generated through convolution and activation: wherein, represents a dynamic weight image, represents a 1x1 convolution layer, represents a convolution layer, represents a rectified linear unit activation function, represents, represents a high-frequency detail image, represents a base image, represents a Sigmoid activation function; By dynamic weight map Fusing the base image and the high-frequency detail map: wherein, denotes the fused image; aligning the fusion result of the previous frame based on inter-frame optical flow: wherein, denotes a fusion reconstruction result of a previous frame, denotes an operator for warping and resampling an image based on an optical flow field, for aligning the previous frame result to a current frame coordinate system, denotes an aligned fusion image of the previous frame, denotes an optical flow field from the t−1th frame to the tth frame; In the fusion process, the current frame is forced to be similar to the aligned previous frame to avoid flickering and texture jumping: where, denotes the temporal consistency loss, denotes the fusion reconstruction result of the current frame, denotes the L1 norm.

8. The method of claim 1, wherein, After each frame of video is reconstructed, the mask update module dynamically updates the motion mask of the next round of iteration according to the current reconstruction quality, comprising: After the current frame is reconstructed, update the motion mask used for the next round of reconstruction: wherein, denotes the new motion mask updated by the mask update module, denotes the mask update module, denotes the fused image, denotes denotes the motion mask down-sampled version on the feature space in the t-th iteration; If the gain of a certain region is lower than the threshold, it is considered as "converged", and the deep layer calculation can be skipped in the subsequent reasoning, realizing block-level adaptive exit. For each block , define a gain score function: wherein, denotes a gain score function of a block , , and respectively denote the amount of improvement in pixel fidelity, perceptual quality and motion consistency of a block ; , , respectively denote the weight coefficients of the amount of improvement in pixel fidelity, perceptual quality and motion consistency. When the block corresponding mask value is set to 0, indicating early exit; otherwise, it is kept as 1, wherein, represents a preset motion intensity threshold.

Images