Video super-resolution reconstruction system and method based on spatiotemporal collaborative enhancement
By using Mamba-based intermediate frame generation and bidirectional differential feature extraction, the spatial and temporal limitations in satellite video spatiotemporal resolution enhancement are resolved, achieving efficient and lightweight spatiotemporal super-resolution reconstruction, generating high-quality intermediate frames, and improving video smoothness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
- Filing Date
- 2026-06-09
- Publication Date
- 2026-07-07
Smart Images
Figure CN122347504A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular relates to a video super-resolution reconstruction system and method based on spatiotemporal co-enhancement. Background Technology
[0002] During the imaging process, the spatial resolution of satellite video is often affected by factors such as atmospheric scattering and extreme weather, which limit further improvements in its spatial resolution. Simultaneously, the temporal resolution of satellite video is also constrained by the limited bandwidth of the sensor, affecting its accurate capture of dynamically changing targets. Faced with these challenges, the remote sensing field urgently needs high-quality satellite video data, which is of great significance for subsequent applications in related fields such as target detection and target tracking. Furthermore, high spatiotemporal resolution satellite video can provide more accurate motion information, helping us analyze the evolution of extreme and transient events. Therefore, improving the spatiotemporal resolution of satellite video has broad application prospects.
[0003] Compared to spatial video super-resolution, spatiotemporal super-resolution requires simultaneous super-resolution in both time and space dimensions to predict non-existent frame pixels and generate high-resolution frame pixels, which is a highly ill-posed problem. Thanks to their powerful nonlinear modeling capabilities, deep neural network models have made significant progress in spatiotemporal super-resolution tasks based on natural videos. Specifically, by merging video super-resolution and video frame interpolation into a joint framework, Kang et al. proposed a deep neural network model for spatiotemporal super-resolution tasks in their ECCV 2020 paper, "Deep space-time video upsampling networks." To better utilize the mutual information between space and time, Haris et al., in their CVPR 2020 paper, "Space-time-aware multiresolution video enhancement," utilized an additional optical flow branch to generate high spatiotemporal resolution videos. Furthermore, to avoid relying on complex convolutional neural network methods, Geng et al. proposed a Swin Transformer-based spatiotemporal video super-resolution network in their CVPR 2022 paper, "RSTT: Real-timespatial temporal transformer for space-time video super-resolution," which naturally integrates spatial and temporal super-resolution modules into a single model. In recent years, Wang et al. proposed the deformable attention network STDAN, which adaptively captures and aggregates spatial and temporal context in dynamic video frames, thereby utilizing more information from neighboring frames for interpolation and reconstruction. Furthermore, in their paper "Space-time super-resolution for satellite video: A joint framework based on multi-scale spatial-temporal transformer" published in the International Journal of Applied Earth Observation and Geoinformation, Xiao et al. proposed a joint network based on multi-scale spatial-temporal transformer (MSTTNet), which is the first to study the joint super-resolution reconstruction of satellite video in terms of temporal and spatial resolution. This model achieves the prediction of non-existent frames by designing an optical flow and multi-scale deformable convolution deep coupling module; at the same time, it fully utilizes the multi-scale spatial-temporal transformer to fully mine multi-scale information to achieve the overall joint super-resolution process.However, this method fails to fully exploit the contextual information within video frames, and the large number of parameters results from embedding multi-scale blocks into multiple heads for spatiotemporal self-attention computation. Furthermore, research on spatiotemporal super-resolution based on satellite video has been scarce in recent years. Therefore, more in-depth theoretical exploration and methodological innovation are needed to achieve more effective and lightweight spatiotemporal super-resolution methods. Summary of the Invention
[0004] In view of this, the present invention aims to provide a video super-resolution reconstruction system and method based on spatiotemporal collaborative enhancement. The intermediate frame generation unit utilizes the Mamba module to achieve complementary advantages in long-distance modeling, multi-scale perception and fusion, and generates high-quality intermediate frame features. The information mining and aggregation unit adaptively enhances cross-frame features and aggregates global spatiotemporal context through bidirectional propagation and differential computing mechanisms.
[0005] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A video super-resolution reconstruction system based on spatiotemporal co-enhancement includes: a feature and optical flow extraction unit, which extracts initial features from two adjacent low-resolution video frames to obtain a first initial feature and a second initial feature; simultaneously extracts forward optical flow and backward optical flow from the two low-resolution video frames according to the temporal direction; and performs corresponding transformations on the two low-resolution video frames using the forward and backward optical flow to obtain a first optical flow feature and a second optical flow feature; and an intermediate frame generation unit, which inputs the first and second initial features into a Mamba module for intermediate frame modeling, and fuses the obtained features with the first and second optical flow features to obtain... Intermediate frame features; an information mining and aggregation unit sorts and fuses the first initial feature, the second initial feature, and the intermediate frame features to obtain a first sorted feature and a second sorted feature; attention operations are performed on the first sorted feature and the second sorted feature respectively, and the two attention weights are added together and appended to the first sorted feature to obtain the reconstructed feature; a frame reconstruction unit decompresses the obtained reconstructed features to obtain the decompressed reconstructed features corresponding to the two low-resolution video frames, as well as the decompressed reconstructed features of the intermediate frame between the two low-resolution video frames; then the three decompressed reconstructed features are decoded and restored to obtain the high-resolution frame images corresponding to the three features respectively.
[0006] Furthermore, the feature and optical flow extraction unit includes an initial feature extraction module and an optical flow generation and mapping module. In the initial feature extraction module, initial features are extracted simultaneously in the first low-resolution video frame and the second low-resolution video frame using multiple consecutive residual blocks, according to the temporal direction, to obtain the first initial feature and the second initial feature. In the optical flow generation and mapping module, the forward optical flow field of each pixel in the first low-resolution video frame that has been displaced in both the first and second low-resolution video frames is calculated. The second initial feature is transformed using the forward optical flow field to obtain the first optical flow feature corresponding to the first low-resolution video frame. The reverse optical flow field of each pixel in the second low-resolution video frame that has been displaced in both the second and first low-resolution video frames is calculated. The first initial feature is transformed using the reverse optical flow field to obtain the second optical flow feature corresponding to the second low-resolution video frame.
[0007] Furthermore, the process of obtaining optical flow features includes: adding the original absolute coordinates of all pixels in the input features to the corresponding optical flow field to obtain the original image sampling coordinates corresponding to each pixel; normalizing the original image sampling coordinates to [-1,1] and then interpolating the current features to obtain the optical flow features.
[0008] Furthermore, in the Mamba module: after converting the first initial feature and the second initial feature from two dimensions to one dimension respectively, the processed two features are concatenated and fused; after inputting the fused features into multiple consecutive Mamba blocks, the resulting feature dimension is restored to two dimensions to obtain the output feature.
[0009] Furthermore, within each Mamba block: the input features are simultaneously processed using visual state space and dual-scale convolutional attention. The processed two features are then added to the weighted input features to obtain preliminary features. The preliminary features are then processed using a multilayer perceptron, and the processed features are then added to the weighted preliminary features to obtain output features.
[0010] Furthermore, the visual state space processing includes two parallel branches: In the first branch, the input features are expanded using a linear mapping layer, and the expanded features are then continuously processed through depthwise convolution, SiLU activation, and selective scanning; In the second branch, the input features are expanded using a linear mapping layer, and SiLU activation is performed on the expanded features; The output features of the two branches are multiplied and then input into the linear mapping layer for channel number recovery, generating output features with the same shape as the input features.
[0011] Furthermore, the dual-scale convolutional attention processing includes: performing depthwise separable convolution operations on the input features at multiple scales; concatenating the resulting multi-scale convolutional features and then performing average pooling and max pooling; concatenating and fusing the features after the two pooling operations and then splitting them into channels; multiplying the split features with the corresponding multi-scale convolutional features; adding the multiplied features and multiplying the added features with the input features to obtain the output features.
[0012] Furthermore, in the information mining aggregation unit: double sorting and feature concatenation are performed in the order of the first initial feature, intermediate frame features, and second initial feature to obtain the first sorted feature and the second sorted feature; the first sorted feature and the second sorted feature are differentially concatenated to obtain their respective differential features; simultaneously, pooling upsampling is performed on the first sorted feature and the second sorted feature to obtain their respective pooled features; for the first sorted feature and the second sorted feature, each feature is fused with its corresponding pooled feature and differential feature, and then channel attention operation is performed on the fused features to extract the channel weights from the two fused features; after adding the two channel weights, the resulting weight value is appended to the first sorted feature or the second sorted feature for feature enhancement to obtain the intermediate frame reconstruction feature.
[0013] A video super-resolution reconstruction method based on spatiotemporal co-enhancement includes: S1: Obtain the real image dataset and degrade the real image dataset to obtain the corresponding low-resolution image set. The low-resolution image set and the real image dataset together form the training set. S2: Construct a video super-resolution reconstruction system based on spatiotemporal collaborative enhancement as provided in this invention; S3: Use the training set obtained in step S1 to train the video super-resolution reconstruction system constructed in step S2 to obtain the video super-resolution reconstruction model. S4: Input the low-resolution image to be reconstructed into the video super-resolution reconstruction model obtained in step S3 to obtain the predicted reconstructed image.
[0014] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention presents a video super-resolution reconstruction system and method based on spatiotemporal co-enhancement. By combining Mamba-based intermediate frame modeling, bidirectional temporal difference feature extraction, and multi-scale feature fusion enhancement, it effectively achieves one-stage spatiotemporal super-resolution reconstruction. Compared with existing optimal one-stage spatiotemporal super-resolution methods, the proposed method yields clearer super-resolution videos with fewer model parameters. Furthermore, by restoring and reconstructing intermediate frame images between adjacent low-resolution video frames, this invention increases the video frame density (e.g., if the original video frames are captured every 1 second, resulting in 100 frames, this invention completes the 0.5-second frame, bringing the reconstructed video to 200 frames), thus improving video smoothness and compensating for missing information caused by the acquisition interval of the shooting equipment. Attached Figure Description
[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic diagram of the overall video super-resolution reconstruction system based on spatiotemporal collaborative enhancement as described in the embodiments of the present invention; Figure 2 A schematic diagram illustrating the data processing process of the Mamba module as described in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the visual state space processing process described in the embodiments of the present invention; Figure 4 A schematic diagram illustrating the dual-scale convolutional attention processing process described in the embodiments of the present invention; Figure 5 A schematic flowchart of the video super-resolution reconstruction method based on spatiotemporal collaborative enhancement as described in the embodiments of the present invention; Figure 6 This is a schematic diagram illustrating the comparative results of an embodiment of the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.
[0017] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this invention can be combined with each other. In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0018] The invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0019] like Figure 1 As shown in the embodiment of the present invention, the video super-resolution reconstruction system based on spatiotemporal collaborative enhancement includes a feature and optical flow extraction unit, an intermediate frame generation unit, an information mining and aggregation unit, and a frame reconstruction unit. The feature and optical flow extraction unit extracts initial features from two adjacent low-resolution video frames to obtain a first initial feature and a second initial feature; simultaneously, it extracts forward optical flow and backward optical flow from the two low-resolution video frames according to the temporal direction; using the forward and backward optical flow, it performs corresponding transformations on the two low-resolution video frames to obtain a first optical flow feature and a second optical flow feature. The intermediate frame generation unit inputs the first and second initial features into a Mamba module for intermediate frame modeling, and fuses the obtained features with the first and second optical flow features to obtain intermediate frame features. The information mining and aggregation unit sorts and fuses the first and second initial features with the intermediate frame features to obtain a first sorted feature and a second sorted feature; it performs attention operations on the first and second sorted features respectively, and then adds the two attention weights together and appends them to the first sorted feature to obtain the reconstructed features. The frame reconstruction unit decompresses the obtained reconstruction features to obtain the decompressed reconstruction features corresponding to the two low-resolution video frames, as well as the decompressed reconstruction features of the intermediate frame between the two low-resolution video frames; then, it decodes and recovers the three decompressed reconstruction features to obtain the high-resolution frame images corresponding to the three features respectively.
[0020] In some embodiments, the feature and optical flow extraction unit includes an initial feature extraction module and an optical flow generation mapping module. In the initial feature extraction module, initial features are extracted simultaneously in the first low-resolution video frame and the second low-resolution video frame through multiple consecutive residual blocks according to the time flow, thereby obtaining the first initial feature and the second initial feature.
[0021] In this embodiment of the invention, the initial feature extraction module uses five consecutive residual blocks to extract features from adjacent low-resolution video frames. Second low resolution video frame Initial feature extraction is performed to obtain the first initial feature. With the second initial feature Each residual block consists of a convolution + activation function + convolution, and residual connections, which captures low-level detailed features, thereby achieving preliminary feature extraction.
[0022] In the optical flow generation and mapping module, the forward optical flow field of each pixel in the first low-resolution video frame that has been displaced in the first low-resolution video frame and the second low-resolution video frame is calculated; the forward optical flow field is used to transform the second initial feature to obtain the first optical flow feature corresponding to the first low-resolution video frame; the reverse optical flow field of each pixel in the second low-resolution video frame that has been displaced in the second low-resolution video frame and the first low-resolution video frame is calculated; the reverse optical flow field is used to transform the first initial feature to obtain the second optical flow feature corresponding to the second low-resolution video frame.
[0023] In some embodiments, the process of obtaining optical flow features includes: adding the original absolute coordinates of all pixels in the input features to the corresponding optical flow field to obtain the original image sampling coordinates corresponding to each pixel; normalizing the original image sampling coordinates to [-1,1] and then interpolating the current features to obtain optical flow features.
[0024] In the optical flow generation mapping module provided in this embodiment of the invention: For the first optical flow feature F1: Calculate the forward optical flow field O for each pixel in the first low-resolution video frame that has undergone displacement in the first and second low-resolution video frames. 1-2 ;Utilizing the forward optical flow field O 1-2 For the second initial feature A transformation is performed to obtain the first optical flow feature F1 corresponding to the first low-resolution video frame. The process of obtaining the first optical flow feature F1 specifically includes: transforming the second initial feature... The original absolute coordinates of all pixels and the forward optical flow field O 1-2 The corresponding values are added together to obtain the original image sampling coordinates for each pixel; after normalizing the original image sampling coordinates to [-1, 1], the current second initial feature is then used. Interpolation is performed to obtain the first optical flow feature F1.
[0025] For the second optical flow feature F2: Calculate the reverse optical flow field O for each pixel in the second low-resolution video frame that has undergone displacement in the second low-resolution video frame and the first low-resolution video frame. 2-1 ; Utilizing the reverse optical flow field O 2-1 For the first initial feature A transformation is performed to obtain the second optical flow feature F2 corresponding to the second low-resolution video frame. The process of obtaining the second optical flow feature F2 specifically includes: transforming the first initial feature... The original absolute coordinates of all pixels and the reverse optical flow field O 2-1 The corresponding values are added together to obtain the original image sampling coordinates for each pixel; after normalizing the original image sampling coordinates to [-1, 1], the current first initial feature is then used. Interpolation is performed to obtain the second optical flow feature F2.
[0026] An effective inter-frame modeling process is crucial for generating high-quality intermediate frames. State-space models have gained widespread attention in recent years due to their excellent performance and linear complexity in long sequence modeling. Based on this, the Mamba model was proposed and rapidly developed; however, research on applying the Mamba model to inter-frame modeling is relatively scarce. Therefore, this invention proposes an inter-frame modeling process based on Mamba. In some embodiments, the data processing of the Mamba module includes: converting a first initial feature and a second initial feature from two dimensions to one dimension respectively; concatenating and fusing the processed two features; inputting the fused features into multiple consecutive Mamba blocks; restoring the feature dimension to two dimensions to obtain the output features. The desired inter-frame modeling effect is gradually achieved through multiple Mamba blocks, and the modeled intermediate frame features are obtained using dimension restoration.
[0027] In this embodiment of the invention, the data processing procedure of the Mamba module is as follows: Figure 2 As shown, the first initial feature With the second initial feature After converting each feature from two dimensions to one dimension, the two processed features are concatenated and fused to obtain feature X, which serves as the input to the first Mamba block. After passing through multiple consecutive Mamba blocks, the feature dimension is restored to two dimensions to obtain the output feature F0.
[0028] In some embodiments, the feature processing of each Mamba block includes: simultaneously performing visual state space processing and dual-scale convolutional attention processing on the input features; adding the processed two features to the weighted input features to obtain preliminary features; performing multilayer perceptron processing on the preliminary features; and then adding the processed features to the weighted preliminary features to obtain output features.
[0029] In this embodiment of the invention, the data processing procedure for each Mamba block is as follows: Figure 2 As shown, feature X is obtained by performing dual-scale convolutional attention processing on feature X and layer normalization operation on feature X. LNNext, visual state space processing is performed. The two processed features are added to the input feature X weighted by α to obtain the preliminary features. The preliminary features are then processed by a multilayer perceptron, and the processed features are added to the preliminary features weighted by α to obtain the output features. Here, the weight α is used to control the residual information and realize dynamic feature aggregation.
[0030] In some embodiments, the visual state space processing includes two parallel branches: in the first branch, the input features are expanded using a linear mapping layer, and the expanded features are continuously processed through depthwise convolution, SiLU activation, and selective scanning; in the second branch, the input features are expanded using a linear mapping layer, and SiLU activation is performed on the expanded features; the output features of the two branches are multiplied and then input into the linear mapping layer for channel number recovery to generate output features with the same shape as the input features.
[0031] In this embodiment of the invention, the process of visual state space processing is as follows: Figure 3 As shown, in the first branch of visual state space processing, a linear mapping layer is used to process the input features X. LN Channel expansion is performed, and the expanded features are continuously passed through depthwise separable convolution, SiLU activation, and selective scanning. Then, layer normalization is performed on the features after selective scanning to obtain the output feature X1. The above process is as follows: X1=LN(SSM(SiLU(DWConv(linear(X LN ))))); Where LN represents layer normalization operation, SSM represents selective scan operation, DWConv represents depthwise separable convolution operation, and linear represents linear mapping layer; In the second branch, a linear mapping layer is used to process the input features X. LN Channel expansion is performed, and SiLU activation is applied to the expanded features to obtain output feature X2. The above process is as follows: X2=SiLU(linear(X LN )); After multiplying the output features X1 and X2, the result is input into a linear mapping layer for channel number recovery, generating an output feature Y1 with the same shape as the input features. The above process is as follows: ; in, It represents the Hadamardi (or Hadama) stack.
[0032] Dual-scale convolutional attention processing is mainly used to compensate for multi-scale features neglected in long-range modeling. Its core idea is to capture features from different receptive fields through a dual-branch structure and adaptively fuse them using channel compression and spatial attention gating mechanisms. Specifically, in some embodiments, the dual-scale convolutional attention processing includes: performing depthwise separable convolution operations on the input features at multiple scales; concatenating the resulting multi-scale convolutional features; performing spatial attention operations on the concatenated features to obtain spatial weights corresponding to each scale; attaching the spatial weights to the convolutional features at the corresponding scales; summing the resulting features; and multiplying the summed features by the input features to obtain the output features.
[0033] In this embodiment of the invention, the process of dual-scale convolutional attention processing is as follows: Figure 4 As shown, the process includes: extracting contextual information from the input feature X using depthwise separable convolutions with kernels of sizes 3×3 and 5×5, respectively; then compressing the channel size to half using 1×1 convolutions to obtain feature X' and feature X'', as shown in the following formula: X'=Conv 1×1 (DWConv 3×3 (X)); X''=Conv 1×1 (DWConv 5×5 (X)); Among them, Conv 1×1 DWConv represents a 1×1 convolution. 3×3 and DWConv 5×5 These represent depthwise separable convolutions with kernels of sizes 3×3 and 5×5, respectively. Feature X' and feature X'' are concatenated by channels to form a dual-scale fused feature X. C As shown in the following formula: X C =Concat(X',X''); Wherein, Concat represents the channel concatenation operation; For dual-scale fusion features X C Perform spatial attention operations to obtain spatial weights for each scale. Specifically: for the dual-scale fused feature X... C After performing both average pooling and max pooling in the spatial direction, the feature channels resulting from the two pooling operations are concatenated. Then, a 7×7 convolution operation with sigmoid activation is performed on the concatenated features to obtain the spatial weights attn. The above process is as follows: attn=sigmoid(Conv 7×7 (Concat(P a (X C ),P m (XC )))); Among them, P a and P m Representing average pooling and max pooling in the spatial direction, respectively, Conv 7×7 This indicates a convolution operation with a kernel size of 7×7; The spatial weights attn are divided according to the number of channels and correspondingly attached to features X' and X''. The two features are then added together, and the sum is multiplied by feature X to obtain the output feature Y2. The above process is as follows: ; Where [:,0] and [:,1] represent the spatial weight attn channel partitioning.
[0034] In the intermediate frame generation unit, the output feature F0 is finally fused with the first optical flow feature F1 and the second optical flow feature F2 to obtain the intermediate frame feature F. LR In this embodiment of the invention, the output feature F0 is concatenated with the first optical flow feature F1 and the second optical flow feature F2 through channels, and then a convolution operation is performed on the concatenated features to complete the fusion of the three features, thereby obtaining the intermediate frame feature F. LR .
[0035] The information mining aggregation unit primarily achieves joint optimization of spatiotemporal features through bidirectional temporal differential feature extraction and multi-scale feature fusion enhancement, which can significantly improve the spatiotemporal consistency and detail recovery capability of satellite video. In some embodiments, the feature processing of the information mining aggregation unit includes: performing double sorting and feature concatenation in the order of first initial features, intermediate frame features, and second initial features to obtain first sorted features and second sorted features; differentially concatenating the first sorted features and second sorted features to obtain their respective differential features, and simultaneously performing pooling upsampling on the first sorted features and second sorted features to obtain their respective pooled features; for the first sorted features and second sorted features, fusing each feature with its corresponding pooled features and differential features, and then performing channel attention operation on the fused features to extract the channel weights from the two fused features; adding the two channel weights and then appending the obtained weight value to the first sorted feature or the second sorted feature for feature enhancement to obtain the intermediate frame reconstruction features.
[0036] In this embodiment of the invention, the feature processing procedure of the information mining aggregation unit is as follows: Figure 1 As shown, it specifically includes: The first initial feature Intermediate frame features F LR With the second initial feature Sort and concatenate to obtain forward features (i.e., the first ranking feature); the second initial feature Intermediate frame features F LR With the first initial feature Sort and concatenate to obtain backward features (i.e., the second ranking feature); as shown in the following formula: ; ; Forward features and backward features Perform the same parallel processing on the forward features During the processing: forward features Convolution processing is performed, and then the resulting first feature x1 is processed in two ways: in one way, the first feature x1 undergoes continuous average pooling, convolution, and upsampling operations; in the other way, the first feature x1 is combined with the backward features. The second feature x2 obtained from convolution is subtracted to obtain the first difference feature y1 = x1 - x2; finally, the first difference feature y1, the first feature x1, and the feature after upsampling are concatenated by channels to obtain the first fused feature F. f1 The above process is as follows: ; Upsample represents the upsampling operation; For the first fusion feature F f1 Performing sequential convolutions, channel attention enhancements, convolutions, and sigmoid activations yields the first fusion weight, attn1, as shown in the following equation: attn1=sigmoid(Conv(CAB(Conv(F f1 )))); Wherein, CAB represents channel attention enhancement operation; Backward features Perform the above processing operations, namely: for backward features Convolution processing is performed, followed by two-way processing of the obtained second feature x2: in one processing step, continuous average pooling, convolution, and upsampling operations are performed on the second feature x2; in the other processing step, the difference between the second feature x2 and the first feature x1 is calculated to obtain the second difference feature y2 = x2 - x1; finally, the second difference feature y2, the second feature x2, and the upsampled feature are concatenated to obtain the second fused feature F. f2 ; for the second fusion feature F f2 Perform consecutive convolutions, channel attention enhancements, convolutions, and sigmoid activations to obtain the second fusion weight attn2; Finally, the first fusion weight attn1 and the second fusion weight attn2 are added together, and the resulting weight value is appended to the forward feature. Feature enhancement is performed on the upper part to obtain the reconstructed features of the intermediate frame.
[0037] In this embodiment of the invention, the frame reconstruction unit consists of 20 residual blocks and a pixel reconstruction layer, which refines and optimizes features and improves resolution to obtain the reconstructed high-resolution video frame corresponding to the first low-resolution video frame. The reconstructed high-resolution video frame corresponding to the second low-resolution video frame And reconstructing high-resolution video frames from the intermediate frames of two low-resolution video frames. Pixel reassembly is a classic upsampling method that can effectively enlarge a scaled-down feature map. It is currently widely used in computer vision-related fields such as super-resolution and image segmentation.
[0038] This invention also provides a video super-resolution reconstruction method based on spatiotemporal co-enhancement, combining... Figure 1 and Figure 5 ,include: S1: Obtain the real image dataset and degrade it to obtain the corresponding low-resolution image set. The low-resolution image set and the real image dataset together form the training set.
[0039] The real image dataset in this embodiment of the invention comes from video sequences taken by the Jilin-1 satellite. Data augmentation operations such as random rotation and flipping are performed on the real image dataset and its corresponding low-resolution image set to obtain the training set, thereby increasing the diversity of the training set.
[0040] S2: Construct a video super-resolution reconstruction system based on spatiotemporal collaborative enhancement as provided in this invention.
[0041] S3: Use the training set obtained in step S1 to train the video super-resolution reconstruction system constructed in step S2 to obtain the video super-resolution reconstruction model.
[0042] The training process adopts an end-to-end approach, training the model by inputting a complete training set. In this embodiment of the invention, the following loss function is used for training: ; Where L represents the loss function, This represents the actual high-resolution video frame corresponding to the first low-resolution video frame, and ε is an empirical parameter used to prevent numerical instability and help stabilize the training process.
[0043] Furthermore, in this embodiment of the invention, the Adam optimizer is used for training, with an initial learning rate set to 1e-4. The training lasts for 100 epochs, and the learning rate decays to 1 / 10 of the initial learning rate at the 50th epoch.
[0044] S4: Input the low-resolution image to be reconstructed into the video super-resolution reconstruction model obtained in step S3 to obtain the predicted reconstructed image.
[0045] To verify the effectiveness of the video super-resolution reconstruction system and method provided by this invention, comparative experiments were also conducted in the embodiments of this invention. Specifically, the method provided by this invention was compared with current state-of-the-art models MSTTNet and STDAN under the same conditions. The comparison results are as follows: Figure 6 As shown, Figure 6 In this model, odd-numbered frames represent the original input frames, even-numbered frames represent newly inserted video frames, and all frames represent the complete super-resolution result containing both odd and even-numbered frames. Furthermore, this embodiment uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation metrics. PSNR is the most commonly used image quality evaluation metric, used to measure the quality of the compressed image; its unit is usually decibels (dB), and a higher value indicates better image quality. Structural Similarity is a full-reference image quality evaluation metric that measures image similarity from three aspects: brightness, contrast, and structure. Its value ranges from [0,1], with values closer to 1 indicating less image distortion and better quality. It can be seen that the method provided by this invention achieves better results in both PSNR and SSIM compared to MSTTNet and STDAN, indicating that the method provided by this invention can effectively reconstruct video. At the same time, the system and method provided by this invention have low parameter computation requirements, allowing them to be embedded into hardware platforms with low computational cost, offering a lightweight advantage and facilitating engineering implementation.
[0046] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.
[0047] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A video super-resolution reconstruction system based on spatiotemporal collaborative enhancement, characterized in that, include: The feature and optical flow extraction unit extracts initial features from two adjacent low-resolution video frames to obtain a first initial feature and a second initial feature; at the same time, it extracts forward optical flow and backward optical flow from the two low-resolution video frames according to the time flow direction; and it performs corresponding transformation on the two low-resolution video frames using the forward optical flow and backward optical flow to obtain a first optical flow feature and a second optical flow feature. The intermediate frame generation unit inputs the first initial feature and the second initial feature into the Mamba module to perform intermediate frame modeling, and fuses the obtained features with the first optical flow feature and the second optical flow feature to obtain the intermediate frame feature; The information mining aggregation unit sorts and fuses the first initial feature, the second initial feature, and the intermediate frame feature to obtain the first sorted feature and the second sorted feature. Attention operations are performed on the first ranking feature and the second ranking feature respectively. The two attention weights are then added together and appended to the first ranking feature to obtain the reconstructed feature. The frame reconstruction unit decompresses the obtained reconstruction features to obtain the decompressed reconstruction features corresponding to the two low-resolution video frames, as well as the decompressed reconstruction features of the intermediate frame between the two low-resolution video frames; then it decodes and recovers the three decompressed reconstruction features to obtain the high-resolution frame images corresponding to the three features respectively.
2. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 1, characterized in that, The feature and optical flow extraction unit includes an initial feature extraction module and an optical flow generation mapping module, wherein: In the initial feature extraction module, initial features are extracted simultaneously in the first low-resolution video frame and the second low-resolution video frame through multiple consecutive residual blocks according to the time flow, so as to obtain the first initial feature and the second initial feature. In the optical flow generation and mapping module, the forward optical flow field of each pixel in the first low-resolution video frame that has been displaced in the first low-resolution video frame and the second low-resolution video frame is calculated; the forward optical flow field is used to transform the second initial feature to obtain the first optical flow feature corresponding to the first low-resolution video frame; the reverse optical flow field of each pixel in the second low-resolution video frame that has been displaced in the second low-resolution video frame and the first low-resolution video frame is calculated; the reverse optical flow field is used to transform the first initial feature to obtain the second optical flow feature corresponding to the second low-resolution video frame.
3. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 2, characterized in that, The process of obtaining optical flow features includes: adding the original absolute coordinates of all pixels in the input features to the corresponding optical flow field to obtain the original image sampling coordinates corresponding to each pixel; normalizing the original image sampling coordinates to [-1,1] and then interpolating the current features to obtain the optical flow features.
4. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 1, characterized in that, In the Mamba module: After converting the first and second initial features from two dimensions to one dimension respectively, the processed two features are spliced and fused together. After inputting the fused features into multiple consecutive Mamba blocks, the resulting feature dimensions are restored to two dimensions to obtain the output features.
5. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 4, characterized in that, In each Mamba block: The input features are simultaneously processed by visual state space and dual-scale convolutional attention. The processed two features are then added to the weighted input features to obtain the preliminary features. The initial features are processed by a multilayer perceptron, and then the processed features are added to the weighted initial features to obtain the output features.
6. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 5, characterized in that, The process of visual state space processing includes two parallel branches: In the first branch, the input features are channel-expanded using a linear mapping layer, and the expanded features are then continuously processed through depthwise convolution, SiLU activation, and selective scanning. In the second branch, a linear mapping layer is used to expand the input features through channels, and the expanded features are then activated using SiLU. After multiplying the output features of the two branches, the product is input into a linear mapping layer to restore the number of channels, generating output features with the same shape as the input features.
7. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 5, characterized in that, The process of dual-scale convolutional attention processing includes: The input features are subjected to depthwise separable convolution operations at multiple scales. The resulting multi-scale convolutional features are then concatenated and then subjected to average pooling and max pooling. The features obtained from the two pooling methods are concatenated and fused, then channel splitting is performed. The split features are then multiplied with the corresponding multi-scale convolutional features. The multiplied features are added together, and the added features are multiplied by the input features to obtain the output features.
8. The video super-resolution reconstruction system based on spatiotemporal collaborative enhancement according to claim 1, characterized in that, In the information mining aggregation unit: The first initial feature, the intermediate frame feature, and the second initial feature are sorted and concatenated to obtain the first sorted feature and the second sorted feature. The first ranking feature and the second ranking feature are differentially concatenated to obtain their respective differential features. At the same time, the first ranking feature and the second ranking feature are pooled and upsampled to obtain their respective pooled features. For the first ranking feature and the second ranking feature, each feature is fused with its corresponding pooling feature and difference feature, and then channel attention operation is performed on the fused features to extract the channel weights in the two fused features. After adding the weights of the two channels, the resulting weight value is added to the first or second sorting feature to enhance the feature and obtain the intermediate frame reconstruction feature.
9. A video super-resolution reconstruction method based on spatiotemporal collaborative enhancement, characterized in that, include: S1: Obtain the real image dataset and degrade the real image dataset to obtain the corresponding low-resolution image set. The low-resolution image set and the real image dataset together form the training set. S2: Construct a video super-resolution reconstruction system based on spatiotemporal collaborative enhancement as described in any one of claims 1 to 8; S3: Use the training set obtained in step S1 to train the video super-resolution reconstruction system constructed in step S2 to obtain the video super-resolution reconstruction model. S4: Input the low-resolution image to be reconstructed into the video super-resolution reconstruction model obtained in step S3 to obtain the predicted reconstructed image.