Video frame reconstruction method and system based on implicit interpolation alignment and feature fusion

Abstract

The invention discloses a video frame reconstruction method and system based on implicit interpolation alignment and feature fusion, and relates to the technical field of super-resolution reconstruction, and the method comprises the steps: obtaining a plurality of continuous to-be-processed video frame images, and correspondingly inputting the video frame images to corresponding video frame reconstruction branches; inputting a current video frame image into a feature extraction module to obtain an extraction feature; inputting the extracted feature and first optical flow information between the current video frame image and the latter two frames into a backward propagation module to obtain a first space-time fusion feature; inputting the first space-time fusion feature, the extraction feature and second optical flow information between the current video frame image and the previous two frames into a forward propagation module to obtain a second space-time fusion feature; inputting the second space-time fusion feature and the current video frame image into a reconstruction module to obtain a high-resolution reconstructed video frame corresponding to the current video frame image; and repeating the process to obtain high-resolution reconstructed video frames corresponding to all video frame images. And super-resolution reconstruction of the satellite video is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of super-resolution reconstruction, and more particularly to a video frame reconstruction method and system based on implicit interpolation alignment and feature fusion. Background Art

[0002] Satellite video is currently widely used in tasks such as geographic monitoring, disaster warning, environmental protection, urban planning, and target identification. Compared to traditional static remote sensing imagery, satellite video provides continuous temporal information, enabling more comprehensive and accurate observation of dynamic changes in land features. However, due to hardware limitations of imaging equipment, data storage, and transmission bandwidth constraints, the resolution and image quality of raw satellite video are typically significantly lower than single-frame images captured by remote sensing satellites. This issue severely impacts the usability of satellite video, blurring target details and edge information, thereby reducing the accuracy of subsequent tasks such as target detection, change detection, and land feature classification, significantly hindering the broader and more in-depth application of satellite video. To overcome this low resolution issue, satellite video super-resolution (SVSR) technology has attracted widespread attention. The successful application of this technology not only significantly improves the visual quality of satellite video but also reduces the reliance on high-cost, high-resolution satellite hardware, providing more efficient data support for a variety of remote sensing tasks.

[0003] Video super-resolution technology has advanced significantly in recent years, and existing methods have significantly improved SVSR performance. However, these methods still face several shortcomings in practical remote sensing applications. First, during the multi-frame alignment process, many methods rely on explicit interpolation methods (such as bilinear interpolation) to map pixels between frames. Although these interpolation methods have good spatial structure preservation capabilities, their low-order reconstruction operations often result in oversmoothing of pixel values, weakening or even losing a large amount of high-frequency detail in the image, thus limiting the clarity and quality of the final reconstructed image. Second, existing methods typically use simple weighting or convolution operations to fuse spatiotemporal information, which is difficult to effectively address the inconsistency of spatiotemporal features caused by occlusion, disappearance, or scale changes during target motion. This fusion strategy easily merges irrelevant or erroneous information from multiple frames, introducing interference signals that affect the reconstruction quality. Finally, many current SVSR methods, while pursuing high accuracy, come with high computational overhead. Due to the wide coverage and high information density of remote sensing images, models must process large amounts of data during inference, resulting in significant consumption of storage and computing resources, making them difficult to adapt to resource-constrained edge computing or real-time applications.

[0004] Therefore, how to retain image details and enhance the ability to model complex motion during the alignment process, and improve the robustness and accuracy of multi-frame video frame fusion, while reducing the parameter scale, and thus achieve a balance between the accuracy and parameter amount of satellite video super-resolution reconstruction, is a problem that technicians in this field urgently need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a video frame reconstruction method and system based on implicit interpolation alignment and feature fusion, which improves the robustness and accuracy of multi-frame video frame fusion, thereby realizing super-resolution reconstruction of satellite video.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The video frame reconstruction method based on implicit interpolation alignment and feature fusion includes:

[0008] Obtaining a plurality of consecutive video frame images to be processed and inputting them into corresponding video frame reconstruction branches;

[0009] Based on the current video frame image input to the feature extraction module, the extracted features are obtained;

[0010] Based on the extracted features and the first optical flow information between the current video frame image and the next two frames, the first optical flow information is input into a backward propagation module to obtain a first spatiotemporal fusion feature;

[0011] Based on the first spatiotemporal fusion feature, the extracted feature, and the second optical flow information between the current video frame image and the previous two frames, the first optical flow information is input into the forward propagation module to obtain a second spatiotemporal fusion feature;

[0012] Based on the second spatiotemporal fusion feature and the current video frame image, a reconstruction module is inputted to obtain a high-resolution reconstructed video frame corresponding to the current video frame image;

[0013] Repeat the above process to obtain the high-resolution reconstructed video frames corresponding to all the video frame images.

[0014] Preferably, the feature extraction module includes: a first convolutional layer and a plurality of first residual convolutional units connected in sequence;

[0015] The previous video frame image is input into the first convolutional layer to obtain initial features;

[0016] sequentially inputting the initial features into a plurality of the first residual convolution units to obtain the processed features;

[0017] The first residual convolution unit includes: a second convolution layer and a third convolution layer;

[0018] The input features are sequentially input into the second convolutional layer and the third convolutional layer to obtain deep features;

[0019] The deep features are fused with the input features to obtain output features.

[0020] Preferably, the back propagation module includes: a first implicit interpolation alignment unit, a second residual convolution unit and a first spatiotemporal feature fusion unit;

[0021] The extracted features and the first optical flow information are input into the first implicit interpolation alignment unit to obtain a first fused alignment feature;

[0022] The extracted features are input into the second residual convolution unit to obtain a first intermediate feature;

[0023] The first intermediate feature and the first fused alignment feature are input into the first spatiotemporal feature fusion unit to obtain the first spatiotemporal fusion feature.

[0024] Preferably, the first implicit interpolation alignment unit includes: a first implicit interpolation alignment sub-unit, a second implicit interpolation alignment sub-unit and a first deformable alignment unit;

[0025] Obtaining the spatiotemporal fusion features of the next frame of the video frame image based on the current video frame image after the next frame of the video frame image is processed by the back propagation module;

[0026] The extracted features, the first optical flow information between the current video frame image and the next video frame image, and the next frame spatiotemporal fusion features are input into the first implicit interpolation alignment subunit to obtain a first alignment feature;

[0027] Obtaining the spatiotemporal fusion features of the next two frames of video frame images based on the processing of the next two frames of video frame images by the back propagation module;

[0028] The extracted features, the first optical flow information between the current video frame image and the next two video frame images, and the next two frames of spatiotemporal fusion features are input into the second implicit interpolation alignment subunit to obtain a second alignment feature;

[0029] The first alignment feature and the second alignment feature are input into the first deformable alignment unit to obtain the first fused alignment feature.

[0030] Preferably, the first spatiotemporal feature fusion unit includes: a fourth convolution layer, a fifth convolution layer, a Sigmoid function layer, a pyramid unit, a first upsampling unit and a second upsampling unit;

[0031] The first intermediate feature and the first fused alignment feature are partially input into the fourth convolutional layer and the fifth convolutional layer, and a first processing feature and a second processing feature are obtained correspondingly;

[0032] The first processing feature and the second processing feature are fused and input into the Sigmoid function layer to obtain a temporal attention weight;

[0033] Obtain a first fused feature based on the fusion of the temporal attention weight and the first fused alignment feature;

[0034] Inputting the first fusion feature into the pyramid unit to obtain a first scale feature, a second scale feature, and a third scale feature;

[0035] The third scale feature is input into the first upsampling unit to obtain a first process feature;

[0036] The first process feature and the second scale feature are fused to obtain a second fused feature;

[0037] The second fusion feature is input into the second upsampling unit to obtain a second process feature;

[0038] The second process feature is fused with the first scale feature to obtain a third fused feature;

[0039] The first spatiotemporal fusion feature is obtained by fusing the third fusion feature with the second process feature.

[0040] Preferably, the forward propagation module includes: a second implicit interpolation alignment unit, a third residual convolution unit and a second spatiotemporal feature fusion unit;

[0041] The first spatiotemporal fusion feature and the second optical flow information are input into the second implicit interpolation alignment unit to obtain a second fusion alignment feature;

[0042] The first spatiotemporal fusion feature and the extracted feature are concatenated and input into the third residual convolution unit to obtain a second intermediate feature;

[0043] The second intermediate feature and the second fused alignment feature are input into the second spatiotemporal feature fusion unit to obtain the second spatiotemporal fusion feature.

[0044] Preferably, the second implicit interpolation alignment unit includes: a third implicit interpolation alignment sub-unit, a fourth implicit interpolation alignment sub-unit and a second deformable alignment unit;

[0045] Obtaining a previous frame of spatiotemporal fusion features based on a previous frame of video frame image of the current video frame image after being processed by the forward propagation module;

[0046] The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous video frame image, and the spatiotemporal fusion feature of the previous frame are input into the third implicit interpolation alignment subunit to obtain a third alignment feature;

[0047] Based on the two video frame images above the current video frame image, the two frames of spatiotemporal fusion features are obtained after being processed by the forward propagation module;

[0048] The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous two video frame images, and the spatiotemporal fusion features of the previous two frames are input into the fourth implicit interpolation alignment subunit to obtain a fourth alignment feature;

[0049] The third alignment feature and the fourth alignment feature are input into the second deformable alignment unit to obtain the second fused alignment feature.

[0050] Preferably, the first alignment feature acquisition method is:

[0051] Obtaining a first pixel displacement in the x and y directions between a current video frame image and a next video frame image based on the first optical flow information;

[0052] Obtaining a next frame window area based on the first pixel displacement and the next frame spatiotemporal fusion features;

[0053] Obtaining a first query pixel based on the extracted features;

[0054] obtaining a first coordinate code of the first query pixel based on the first pixel displacement;

[0055] Obtaining a first Q value based on the first query pixel and the first coordinate code input into a query information coordinate network;

[0056] Obtaining a second coordinate code of the next frame window area based on the next frame window area;

[0057] Based on the next frame window area and the second coordinate code, they are input into the key information coordinate network and the value information coordinate network respectively, and a first K value and a first V value are obtained accordingly;

[0058] Obtaining an alignment result for each pixel in the extracted feature based on the first Q value, the first K value, and the first V value;

[0059] The first alignment feature is obtained based on the alignment results of all pixels.

[0060] Preferably, the reconstruction module comprises: a fourth residual convolution unit, a pixel shuffling unit and a third upsampling unit;

[0061] The second spatiotemporal fusion unit is sequentially input into the fourth residual convolution unit and the pixel shuffling unit to obtain a high-resolution frame image;

[0062] The current frame image is input to the third upsampling unit for processing and then fused with the high-resolution frame image to obtain the high-resolution reconstructed video frame.

[0063] A video frame reconstruction system based on implicit interpolation alignment and feature fusion includes: a data acquisition module, a first feature extraction module, a second feature extraction module, a third feature extraction module and a reconstruction output module;

[0064] The data acquisition module is used to acquire a plurality of consecutive video frame images to be processed and input them into corresponding video frame reconstruction branches;

[0065] The first feature extraction module is configured to obtain extracted features based on the current video frame image input to the feature extraction module;

[0066] The second feature extraction module is configured to input the extracted features and the first optical flow information between the current video frame image and the next two frames into the backward propagation module to obtain a first spatiotemporal fusion feature;

[0067] The third feature extraction module is configured to input the first spatiotemporal fusion feature, the extracted feature, and the second optical flow information between the current video frame image and the previous two frames into the forward propagation module to obtain a second spatiotemporal fusion feature;

[0068] The reconstruction output module is used to input the second spatiotemporal fusion feature and the current video frame image into the reconstruction module to obtain a high-resolution reconstructed video frame corresponding to the current video frame image; repeat the above process to obtain the high-resolution reconstructed video frames corresponding to all the video frame images.

[0069] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a video frame reconstruction method and system based on implicit interpolation alignment and feature fusion. The present invention introduces an alignment module based on implicit interpolation. By using continuous implicit functions to more flexibly map pixels between frames, it effectively alleviates the image smoothing and information loss caused by traditional explicit interpolation methods, effectively retains image details and enhances the modeling ability of complex motion. Secondly, considering that problems such as target occlusion and scale changes may lead to inconsistent information in multiple frames, the present invention designs a spatiotemporal information fusion module, which combines inter-frame saliency modeling and temporal attention mechanism to accurately capture key dynamic information between multiple frames, and shows stronger adaptability and fusion effect in typical remote sensing complex environments with occlusion, illumination changes and scale inconsistency, thereby improving the robustness and accuracy of multi-frame fusion. Finally, in response to the computational bottleneck in the remote sensing data processing process, the present invention systematically optimizes the network structure and module design to reduce the parameter scale and computational cost while ensuring performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0071] Figure 1 This is a flow chart of the video frame reconstruction method based on implicit interpolation alignment and feature fusion provided by the present invention.

[0072] Figure 2 This is a schematic diagram of the video frame reconstruction model structure provided by the present invention.

[0073] Figure 3 This is a schematic diagram of the backward propagation module structure provided by the present invention.

[0074] Figure 4 Schematic diagram of the structure of the first spatiotemporal feature fusion unit provided by the present invention.

[0075] Figure 5 This is a schematic diagram comparing the effects of reconstructing runway lines provided by the present invention.

[0076] Figure 6 This is a schematic diagram comparing the effects of reconstructed building roofs provided by the present invention.

[0077] Figure 7 Schematic diagram of the video frame reconstruction system structure based on implicit interpolation alignment and feature fusion provided by the present invention. DETAILED DESCRIPTION

[0078] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0079] Example 1

[0080] like Figure 1 As shown, an embodiment of the present invention discloses a video frame reconstruction method based on implicit interpolation alignment and feature fusion, comprising:

[0081] Obtaining a plurality of consecutive video frame images to be processed and inputting them into corresponding video frame reconstruction branches;

[0082] Based on the current video frame image input to the feature extraction module, the extracted features are obtained;

[0083] Based on the extracted features and the first optical flow information between the current video frame image and the next two frames, the first spatiotemporal fusion features are obtained by inputting them into the backward propagation module;

[0084] Based on the first spatiotemporal fusion feature, the extracted feature and the second optical flow information between the current video frame image and the previous two frames, the first spatiotemporal fusion feature is input into the forward propagation module to obtain the second spatiotemporal fusion feature;

[0085] Based on the second spatiotemporal fusion feature and the current video frame image, a high-resolution reconstructed video frame corresponding to the current video frame image is input into the reconstruction module;

[0086] Repeat the above process to obtain high-resolution reconstructed video frames corresponding to all video frame images.

[0087] Example 2

[0088] An embodiment of the present invention discloses a video frame reconstruction method based on implicit interpolation alignment and feature fusion, comprising:

[0089] A plurality of continuous video frame images to be processed are obtained and input into corresponding video frame reconstruction branches.

[0090] Preferably, the video frame image to be processed in this embodiment is: a low-resolution satellite video frame image, set as x t-n ,...,x t ,...,x t+n , where x t Represents the video frame image at the current time step t, that is, the current video frame image, x t-n Represents the current video frame image x tThe video frame image of the first n frames, x t+n Represents the current video frame image x t The video frame image after n frames.

[0091] Preferably, Figure 2 As shown in the figure, the continuous video frame images to be processed are input to the video frame reconstruction branch, which includes: feature extraction module, backward propagation module, forward propagation module and reconstruction module; the video frame reconstruction branch is the video frame reconstruction model, which adopts the classic second-order bidirectional cyclic video super-resolution network architecture.

[0092] Based on the current video frame image input to the feature extraction module, the extracted features are obtained.

[0093] Preferably, the feature extraction module includes: a first convolutional layer and a plurality of first residual convolutional units connected in sequence;

[0094] The previous video frame image is input into the first convolutional layer to obtain the initial features;

[0095] Based on the initial features, the features are sequentially input into multiple first residual convolution units to obtain processed features;

[0096] The first residual convolution unit includes: a second convolution layer and a third convolution layer;

[0097] The input features are sequentially input to the second and third convolutional layers to obtain deep features;

[0098] The deep features are fused with the input features to obtain the output features.

[0099] Preferably, the data processing process of the feature extraction module is illustrated here by taking the previous video frame image as an example, and the processing process of the video frame images at other time steps in the feature extraction module is the same as this part, which will not be described in detail here.

[0100] Based on the extracted features and the first optical flow information between the current video frame image and the next two frames, the information is input into the backward propagation module to obtain the first spatiotemporal fusion features.

[0101] Preferably, Figure 3 As shown, the back propagation module includes: a first implicit interpolation alignment unit, a second residual convolution unit and a first spatiotemporal feature fusion unit;

[0102] The extracted features and the first optical flow information are input into a first implicit interpolation alignment unit to obtain a first fused alignment feature;

[0103] The extracted features are input into the second residual convolution unit to obtain the first intermediate features;

[0104] The first intermediate feature and the first fused alignment feature are input into the first spatiotemporal feature fusion unit to obtain the first spatiotemporal fusion feature.

[0105] Preferably, the first implicit interpolation alignment unit includes: a first implicit interpolation alignment sub-unit, a second implicit interpolation alignment sub-unit and a first deformable alignment unit;

[0106] Based on the current video frame image, the next video frame image is processed by the back propagation module to obtain the next frame of spatiotemporal fusion features;

[0107] The extracted features, the first optical flow information between the current video frame image and the next video frame image, and the next frame spatiotemporal fusion features are input into the first implicit interpolation alignment subunit to obtain the first alignment feature;

[0108] Based on the two video frames under the current video frame image, the spatiotemporal fusion features of the next two frames are obtained after being processed by the back propagation module;

[0109] The extracted features, the first optical flow information between the current video frame image and the next two video frame images, and the spatiotemporal fusion features of the next two frames are input into the second implicit interpolation alignment subunit to obtain the second alignment features;

[0110] The first alignment feature and the second alignment feature are input into a first deformable alignment unit to obtain a first fused alignment feature.

[0111] Preferably, in this embodiment, the current video frame image is x t , current video frame image x t The next video frame image is x t+1 , current video frame image x t The next two video frames are x t+2 .

[0112] Preferably, the first alignment feature acquisition method is:

[0113] Obtaining a first pixel displacement in the x and y directions between the current video frame image and the next video frame image based on the first optical flow information;

[0114] The next frame window area is obtained based on the first pixel displacement and the next frame spatiotemporal fusion features;

[0115] Obtaining a first query pixel based on the extracted features;

[0116] Obtaining a first coordinate code of a first query pixel based on the first pixel displacement;

[0117] Obtaining a first Q value based on the first query pixel and the first coordinate code input into the query information coordinate network;

[0118] Obtaining a second coordinate code of the next frame window area based on the next frame window area;

[0119] Based on the next frame window area and the second coordinate code, the key information coordinate network and the value information coordinate network are input respectively to obtain a first K value and a first V value respectively;

[0120] Obtaining an alignment result for each pixel in the extracted feature based on the first Q value, the first K value, and the first V value;

[0121] Based on the alignment results of all pixels, a first alignment feature is obtained.

[0122] Preferably, the optical flow is divided into integer parts (d x ,d y ), that is, the integer part of the pixel displacement in the x and y directions from the next frame image (reference frame image) to the current video frame image, and the corresponding decimal part (f x ,f y ):

[0123] OF(x,y)=(d x ,d y )+(f x ,f y );

[0124] The integer part is used to select the window position in the reference frame, while the decimal part reflects the relative position between the current frame pixel and the pixel in the window and is used for position encoding.

[0125] Preferably, based on the extracted feature x t 1 Get the first query pixel X t =x t 1 [x, y]; obtain the first coordinate code P of the first query pixel based on the first pixel displacement; t =γ((d x ,d y ) / 2w), where w represents the width of the window.

[0126] Preferably, based on the first pixel displacement (d x ,d y ) and the next frame spatiotemporal fusion feature x r Get the next frame window area W r [i,j]=x r [x+d x +i,y+d y +j], where i and j are used to iterate over each pixel in the window;

[0127] The second coordinate code of the next frame window area Where γ represents the position encoding function, and the position encoding γ(p) is obtained by mapping the two-dimensional coordinate p onto a 4D hypersphere:

[0128] γ(p)=[[sin(ωp),cos(ωp)],...[sin(ω D-1 p), cos(ω D-1 p)]];

[0129] Where ω represents the angular frequency, and D is used to control the frequency from ω to ω. D-1 The number of frequency bands.

[0130] Preferably, based on the first query pixel X t and the first coordinate code P t Input to query information coordinate network F q , get the first Q value: Q = F q (X t +P t ); Based on the next frame window area W r and the second coordinate code P r Input into the key information coordinate network F K Sum value information coordinate network F V , corresponding to the first K value K and the first V value V:

[0131] K=F K (W r +P r );

[0132] V=F V (W r +P r );

[0133] Based on the first Q value Q, the first K value K and the first V value V, the alignment result x of each pixel in the extracted feature is obtained. ta [x,y]:

[0134]

[0135] Among them, softmax() is used to calculate the current frame extraction feature x t 1 The pixel at position [x,y] in and the window area in the next frame image The correlation between them.

[0136] Preferably, the second alignment feature acquisition method is the same as the first alignment feature acquisition method, both of which are:

[0137] Obtaining a second pixel displacement in the x and y directions between the current video frame image and the next two frames of image based on the first optical flow information;

[0138] The window areas of the next two frames are obtained based on the second pixel displacement and the spatiotemporal fusion features of the next two frames;

[0139] Obtaining a second query pixel based on the extracted features;

[0140] Obtaining a third coordinate code of the second query pixel based on the second pixel displacement;

[0141] Based on the second query pixel and the second coordinate encoding input to the query information coordinate network, a second Q value is obtained;

[0142] Obtaining a fourth coordinate code of the next two frame window area based on the next two frame window area;

[0143] Based on the next two frame window areas and the fourth coordinate code, they are input into the key information coordinate network and the value information coordinate network respectively, and the second K value and the second V value are obtained accordingly;

[0144] Obtaining an alignment result for each pixel in the extracted feature based on the second Q value, the second K value, and the second V value;

[0145] Based on the alignment results of all pixels, a second alignment feature is obtained.

[0146] Preferably, in existing satellite video super-resolution methods, the interpolation step is often overlooked. Most methods default to using traditional explicit interpolation techniques, such as bilinear interpolation, in their alignment modules. However, the choice of interpolation method has a significant impact on the quality of the alignment results. To address this problem, the present invention proposes to use a coordinate-based neural network, namely a structure combining positional encoding (PE) and a multi-layer perceptron (MLP) (PE-MLP), to perform implicit interpolation.

[0147] Preferably, the last frame of the video frame image does not have the corresponding optical flow information because there is no next frame and the next two frames of images. Its input into the backward propagation module is only the extracted features, wherein the output of the first implicit interpolation alignment unit (IIAM) is empty, and the extracted features are input into the second residual convolution unit to obtain the corresponding intermediate features. The corresponding intermediate features and the empty input are input into the first spatiotemporal feature fusion unit to obtain the corresponding spatiotemporal fusion features.

[0148] Preferably, the penultimate frame of the video frame image does not have the next two frames of images, and only has the first optical flow information. The input to the backward propagation module is the extracted features and the first optical flow information, wherein the first implicit interpolation alignment subunit in the first implicit interpolation alignment unit processes the extracted features, the first optical flow information and the next frame of spatiotemporal fusion features to obtain a corresponding alignment feature, and the second implicit interpolation alignment subunit does not work. Finally, based on a corresponding alignment feature, it is input to the first deformable alignment unit to obtain a corresponding fused alignment feature, the extracted feature is input to the second residual convolution unit to obtain a corresponding intermediate feature, and the corresponding intermediate feature and the fused alignment feature are input to the first spatiotemporal feature fusion unit to obtain a corresponding spatiotemporal fusion feature.

[0149] Preferably, in this embodiment, the second residual convolution unit has the same structure as the first residual convolution unit.

[0150] Preferred, preferred, such as Figure 4 As shown, the first spatiotemporal feature fusion unit includes: a fourth convolutional layer, a fifth convolutional layer, a Sigmoid function layer, a pyramid unit, a first upsampling unit and a second upsampling unit;

[0151] The first intermediate feature and the first fused alignment feature are partially input to the fourth convolutional layer and the fifth convolutional layer to obtain the first processing feature and the second processing feature respectively;

[0152] The first processed feature and the second processed feature are fused and input into the Sigmoid function layer to obtain the temporal attention weight;

[0153] The first fusion feature is obtained by fusing the temporal attention weight and the first fusion alignment feature;

[0154] Based on the first fusion feature input to the pyramid unit, the first scale feature, the second scale feature and the third scale feature are obtained;

[0155] The third scale feature is input into the first upsampling unit to obtain the first process feature;

[0156] The first process feature is fused with the second scale feature to obtain the second fused feature;

[0157] The second fusion feature is input into the second upsampling unit to obtain the second process feature;

[0158] The second process feature is fused with the first scale feature to obtain the third fusion feature;

[0159] The third fusion feature is fused with the second process feature to obtain the first spatiotemporal fusion feature.

[0160] Preferably, the fourth convolution layer and the fifth convolution layer both use 3×3 convolution layers, and the first processing feature and the second processing feature are input into the Sigmoid function layer to limit the output to the range of [0,1] to obtain the temporal attention weight, which is used to weight the first fusion alignment feature to obtain the output result of the temporal attention mechanism, namely the first fusion feature.

[0161] In satellite video super-resolution tasks, complex object motion, large-scale changes, and frequent occlusions make it difficult for simple multi-frame fusion strategies to achieve robust and accurate reconstruction results. To address these issues, the present invention designs a spatiotemporal fusion module (SFM), which combines temporal and spatial attention mechanisms to more effectively capture temporal dependencies and spatial structural information across frames, thereby improving reconstruction performance and stability.

[0162] The temporal attention mechanism takes as input the processed features of the current frame and the aligned features of the reference frame. It dynamically adjusts the fusion weights based on the importance of each frame, thereby guiding the network to focus on frames with richer information. Simultaneously, the spatial attention mechanism introduces a pyramid structure, projecting features onto three different spatial resolutions, thereby enhancing the model's ability to perceive and represent multi-scale spatial content. With the synergistic effect of these two attention mechanisms, the proposed fusion module can more robustly handle complex motion, illumination changes, and occlusions in satellite video, providing a stronger information fusion representation and laying the foundation for high-quality reconstruction.

[0163] Preferably, the first scale feature is obtained based on the first fusion feature input to the pyramid unit. Second scale features and third-scale features

[0164] Based on the first spatiotemporal fusion feature, the extracted feature and the second optical flow information between the current video frame image and the previous two frames are input into the forward propagation module to obtain the second spatiotemporal fusion feature.

[0165] Preferably, the forward propagation module includes: a second implicit interpolation alignment unit, a third residual convolution unit and a second spatiotemporal feature fusion unit;

[0166] The first spatiotemporal fusion feature and the second optical flow information are input into a second implicit interpolation alignment unit to obtain a second fusion alignment feature;

[0167] The first spatiotemporal fusion feature and the extracted feature are concatenated and input into the third residual convolution unit to obtain the second intermediate feature;

[0168] The second intermediate feature and the second fused alignment feature are input into the second spatiotemporal feature fusion unit to obtain the second spatiotemporal fusion feature.

[0169] Preferably, the second implicit interpolation alignment unit includes: a third implicit interpolation alignment sub-unit, a fourth implicit interpolation alignment sub-unit and a second deformable alignment unit;

[0170] The spatiotemporal fusion features of the previous frame of the current video frame are obtained after the previous frame of the current video frame is processed by the forward propagation module;

[0171] The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous video frame image, and the spatiotemporal fusion feature of the previous frame are input into the third implicit interpolation alignment subunit to obtain a third alignment feature;

[0172] The spatiotemporal fusion features of the two frames above the current video frame are obtained after the two frames above the current video frame are processed by the forward propagation module;

[0173] The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous two video frame images, and the spatiotemporal fusion features of the previous two frames are input into the fourth implicit interpolation alignment subunit to obtain a fourth alignment feature;

[0174] The third alignment feature and the fourth alignment feature are input into the second deformable alignment unit to obtain a second fused alignment feature.

[0175] Preferably, in this embodiment, the current video frame image is x t , current video frame image x t The previous video frame image is x t-1 , current video frame image x t The last two video frames are x t-2 .

[0176] Preferably, the third alignment feature acquisition method is:

[0177] Obtaining a third pixel displacement in the x and y directions between the current video frame image and the previous two frames of image based on the second optical flow information;

[0178] The window area of ​​the previous two frames is obtained based on the third pixel displacement and the spatiotemporal fusion features of the previous two frames;

[0179] Obtaining a third query pixel based on the first spatiotemporal fusion feature;

[0180] Obtaining a fifth coordinate code of a third query pixel based on the third pixel displacement;

[0181] Obtaining a third Q value based on the third query pixel and the fifth coordinate encoding input to the query information coordinate network;

[0182] Obtaining a sixth coordinate code of the last two frame window area based on the last two frame window area;

[0183] Based on the window areas of the previous two frames and the sixth coordinate code, they are input into the key information coordinate network and the value information coordinate network respectively, and the third K value and the third V value are obtained accordingly;

[0184] Obtaining an alignment result of each pixel in the first spatiotemporal fusion feature based on the third Q value, the third K value, and the third V value;

[0185] Based on the alignment results of all pixels, a third alignment feature is obtained.

[0186] Preferably, the fourth alignment feature is obtained in the same manner as the third alignment feature, and will not be described in detail here.

[0187] Preferably, the third residual convolution unit has the same structure as the above-mentioned first residual convolution unit, and the data processing process is also the same; the second spatiotemporal feature fusion unit structure and data processing process are the same as the above-mentioned first spatiotemporal feature fusion unit, and will not be repeated here one by one.

[0188] Preferably, the first frame of video frame image does not have the previous frame and the previous two frames of image, and there is no corresponding optical flow information. Its input into the forward propagation module is only the first spatiotemporal fusion feature and the extracted feature, wherein the output of the second implicit interpolation alignment unit (IIAM) is empty, and the first spatiotemporal fusion feature and the extracted feature are spliced ​​and input into the third residual convolution unit to obtain the corresponding intermediate feature, and the corresponding intermediate feature and the empty input are input into the second spatiotemporal feature fusion unit to obtain the corresponding spatiotemporal fusion feature.

[0189] Preferably, the second frame of video frame image does not have the previous two frames of images, and only has the second optical flow information. The first spatiotemporal fusion feature, the extracted feature and the second optical flow information are input into the forward propagation module, wherein the third implicit interpolation alignment subunit in the second implicit interpolation alignment unit processes the first spatiotemporal fusion feature, the first optical flow information and the spatiotemporal fusion feature of the previous frame to obtain a corresponding alignment feature, and the fourth implicit interpolation alignment subunit does not work. Finally, based on a corresponding alignment feature, it is input into the second deformable alignment unit to obtain a corresponding fused alignment feature. The first spatiotemporal fusion feature and the extracted feature are spliced ​​and input into the third residual convolution unit to obtain a corresponding intermediate feature. The corresponding intermediate feature and the fused alignment feature are input into the second spatiotemporal feature fusion unit to obtain a corresponding spatiotemporal fusion feature.

[0190] Based on the second spatiotemporal fusion feature and the current video frame image, a reconstruction module is input to obtain a high-resolution reconstructed video frame corresponding to the current video frame image.

[0191] Preferably, the reconstruction module includes: a fourth residual convolution unit, a pixel shuffling unit and a third upsampling unit;

[0192] The second spatiotemporal fusion unit sequentially inputs the data into the fourth residual convolution unit and the pixel shuffling unit to obtain a high-resolution frame image;

[0193] The current frame image is input to the third upsampling unit for processing and then fused with the high-resolution frame image to obtain a high-resolution reconstructed video frame.

[0194] Repeat the above process to obtain high-resolution reconstructed video frames corresponding to all video frame images.

[0195] Preferably, all the video frame images to be processed are processed frame by frame based on the above process to obtain high-resolution reconstructed video frames corresponding to all frames, thereby completing high-quality reconstruction output of the entire low-resolution satellite video.

[0196] Example 3

[0197] To evaluate the effectiveness of the present invention, a satellite video super-resolution dataset was constructed by screening and organizing two publicly available datasets: SATSOT and SAT-MTB. The SATSOT dataset contains video sequences captured by three commercial satellites (Jilin-1, Skybox, and Carbonite-2), while the SAT-MTB dataset contains satellite videos captured by Jilin-1. These two datasets cover a variety of ground targets such as trains, cars, airplanes, and ships, showing a variety of motion patterns and behavioral characteristics, including high-speed and low-speed movement, turning, stillness, occlusion, and objects entering and exiting the field of view.

[0198] Specifically, a total of 172 satellite video clips were cropped from 18 SAT-MTB videos and 54 SATSOT videos. Each clip contains 100 consecutive frames, each with a resolution of 640×640 pixels. Of these clips, 150 were selected for training, and the remaining 22 were used for validation. The original videos were treated as high-resolution images as the ground-truth labels for reconstruction, and the corresponding low-resolution images were generated by 4×bicubic downsampling the original images, with a resolution of 160×160 pixels. These LR-HR image pairs were used to train and evaluate the super-resolution reconstruction performance of the present invention.

[0199] In order to fully verify the effectiveness of the proposed model of the present invention, the present invention is compared with a variety of representative video super-resolution methods, including the traditional Bicubic interpolation method, and BasicVSR, BasicVSR++ and RASVSR based on deep learning. The first two methods are mainly for natural scene videos, while RASVSR is designed for remote sensing scenes. Through cross-domain comparison, the adaptability and superiority of the method of the present invention in remote sensing videos are fully reflected. During the evaluation process, three commonly used image quality evaluation indicators were used: PSNR (peak signal-to-noise ratio), SSIM (structural similarity) and NIQE (natural image quality evaluation). At the same time, the parameter quantity indicator Params was used to evaluate the parameter scale of the model, in units of millions (M, million), and the number of video input frames Frames of different methods was shown. The evaluation results are shown in Table 1:

[0200] Table 1 Evaluation results

[0201] method Bicubic BasicVSR BasicVSR++ RASVSR This method PSNR 35.8943 38.2760 38.4253 38.5895 39.1592 SSIM 0.9299 0.9533 0.9556 0.9557 0.9589 NIQE 7.9213 7.4212 7.3995 7.3799 7.3573 Params(M) / 6.29 7.32 5.39 5.40 Frames 1 15 30 30 30

[0202] Experimental results show that our method achieves the highest values ​​in PSNR and SSIM (39.1592dB and 0.9589 respectively), and the lowest value of 7.3573 in NIQE, which fully demonstrates its comprehensive performance advantages in detail reconstruction, structure preservation and perceptual quality.

[0203] This method accepts 30 consecutive frames as input, significantly more than BasicVSR (15 frames), and can more fully exploit cross-frame spatiotemporal information. It has stronger compensation and reasoning capabilities in the presence of occlusion or weak texture. Comparative methods, however, often struggle to handle the complex multi-scale motion and occlusion problems in remote sensing videos due to frame limitations.

[0204] In comparison with BasicVSR++ and RASVSR, this method improves PSNR by approximately 0.73dB and 0.57dB, respectively, and SSIM by 0.0033 and 0.0032, respectively, while further reducing NIQE. These improvements are attributed to differences in structural design: BasicVSR++ is optimized for natural scenes and has difficulty adapting to perspective changes and scale variations in remote sensing; RASVSR, while lightweight, relies on explicit optical flow alignment and has difficulty handling large-scale non-rigid motion. The implicit interpolation-based alignment module proposed in this invention, combined with a coordinate modeling mechanism, can effectively cope with nonlinear temporal deformations. At the same time, the introduced spatiotemporal fusion module combines temporal attention with a spatial pyramid structure to effectively enhance feature expression capabilities and occlusion robustness.

[0205] In addition, the model parameters of this method are only 5.40M, which is close to RASVSR's 5.39M and much smaller than BasicVSR (6.29M) and BasicVSR++ (7.32M). Even when processing 30 frames of input, it still shows good practicality and deployment potential.

[0206] Qualitative image comparison results are as follows Figure 5-Figure 6 As shown, this method performs better in reconstructing details such as runway lines and building roofs, effectively suppressing artifacts and texture loss, and is suitable for complex applications such as remote sensing video. Overall, this method achieves superior reconstruction quality while maintaining lightweight, demonstrating strong engineering application value and widespread adoption.

[0207] Example 4

[0208] like Figure 7 As shown, the video frame reconstruction system based on implicit interpolation alignment and feature fusion includes: a data acquisition module, a first feature extraction module, a second feature extraction module, a third feature extraction module and a reconstruction output module;

[0209] A data acquisition module is used to acquire a plurality of consecutive video frame images to be processed and input them into corresponding video frame reconstruction branches;

[0210] A first feature extraction module is used to obtain extracted features based on the current video frame image input to the feature extraction module;

[0211] A second feature extraction module is configured to input the extracted features and the first optical flow information between the current video frame image and the next two frames into the backward propagation module to obtain a first spatiotemporal fusion feature;

[0212] A third feature extraction module is configured to obtain a second spatiotemporal fusion feature based on the first spatiotemporal fusion feature, the extracted feature, and the second optical flow information between the current video frame image and the previous two frames, which are input into the forward propagation module;

[0213] The reconstruction output module is used to input the second spatiotemporal fusion feature and the current video frame image into the reconstruction module to obtain a high-resolution reconstructed video frame corresponding to the current video frame image; repeat the above process to obtain high-resolution reconstructed video frames corresponding to all video frame images.

[0214] Preferably, in this embodiment, the implementation process of each functional module corresponds to the content of the above method one by one, and will not be repeated here.

[0215] Example 5

[0216] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0217] Memory for storing computer programs;

[0218] The processor, when used to execute the program stored in the memory, can implement the video frame reconstruction method based on implicit interpolation alignment and feature fusion as in Example 1 or 2.

[0219] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute the video frame reconstruction method based on implicit interpolation alignment and feature fusion in embodiment 1 or 2.

[0220] In addition, the logical instructions in the aforementioned memory can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention.

[0221] The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.

[0222] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0223] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video frame reconstruction method based on implicit interpolation alignment and feature fusion, characterized in that: include: Obtaining a plurality of consecutive video frame images to be processed and inputting them into corresponding video frame reconstruction branches; Based on the current video frame image input to the feature extraction module, the extracted features are obtained; Based on the extracted features and the first optical flow information between the current video frame image and the next two frames, the first optical flow information is input into a backward propagation module to obtain a first spatiotemporal fusion feature; Based on the first spatiotemporal fusion feature, the extracted feature, and the second optical flow information between the current video frame image and the previous two frames, the first optical flow information is input into the forward propagation module to obtain a second spatiotemporal fusion feature; Based on the second spatiotemporal fusion feature and the current video frame image, a reconstruction module is inputted to obtain a high-resolution reconstructed video frame corresponding to the current video frame image; Repeat the above process to obtain the high-resolution reconstructed video frames corresponding to all the video frame images.

2. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 1, characterized in that: The feature extraction module includes: a first convolution layer and a plurality of first residual convolution units connected in sequence; The previous video frame image is input into the first convolutional layer to obtain initial features; sequentially inputting the initial features into a plurality of the first residual convolution units to obtain the processed features; The first residual convolution unit includes: a second convolution layer and a third convolution layer; The input features are sequentially input into the second convolutional layer and the third convolutional layer to obtain deep features; The deep features are fused with the input features to obtain output features.

3. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 1, characterized in that: The back propagation module includes: a first implicit interpolation alignment unit, a second residual convolution unit and a first spatiotemporal feature fusion unit; The extracted features and the first optical flow information are input into the first implicit interpolation alignment unit to obtain a first fused alignment feature; The extracted features are input into the second residual convolution unit to obtain a first intermediate feature; The first intermediate feature and the first fused alignment feature are input into the first spatiotemporal feature fusion unit to obtain the first spatiotemporal fusion feature.

4. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 3 is characterized in that: The first implicit interpolation alignment unit includes: a first implicit interpolation alignment sub-unit, a second implicit interpolation alignment sub-unit and a first deformable alignment unit; Obtaining the spatiotemporal fusion features of the next frame of the video frame image based on the current video frame image after the next frame of the video frame image is processed by the back propagation module; The extracted features, the first optical flow information between the current video frame image and the next video frame image, and the next frame spatiotemporal fusion features are input into the first implicit interpolation alignment subunit to obtain a first alignment feature; Obtaining the spatiotemporal fusion features of the next two frames of video frame images based on the processing of the next two frames of video frame images by the back propagation module; The extracted features, the first optical flow information between the current video frame image and the next two video frame images, and the next two frames of spatiotemporal fusion features are input into the second implicit interpolation alignment subunit to obtain a second alignment feature; The first alignment feature and the second alignment feature are input into the first deformable alignment unit to obtain the first fused alignment feature.

5. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 3, characterized in that: The first spatiotemporal feature fusion unit includes: a fourth convolution layer, a fifth convolution layer, a Sigmoid function layer, a pyramid unit, a first upsampling unit and a second upsampling unit; The first intermediate feature and the first fused alignment feature are partially input into the fourth convolutional layer and the fifth convolutional layer, and a first processing feature and a second processing feature are obtained correspondingly; The first processing feature and the second processing feature are fused and input into the Sigmoid function layer to obtain a temporal attention weight; Obtain a first fused feature based on the fusion of the temporal attention weight and the first fused alignment feature; Inputting the first fusion feature into the pyramid unit to obtain a first scale feature, a second scale feature, and a third scale feature; The third scale feature is input into the first upsampling unit to obtain a first process feature; The first process feature and the second scale feature are fused to obtain a second fused feature; The second fusion feature is input into the second upsampling unit to obtain a second process feature; The second process feature is fused with the first scale feature to obtain a third fused feature; The first spatiotemporal fusion feature is obtained by fusing the third fusion feature with the second process feature.

6. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 1, characterized in that: The forward propagation module includes: a second implicit interpolation alignment unit, a third residual convolution unit and a second spatiotemporal feature fusion unit; The first spatiotemporal fusion feature and the second optical flow information are input into the second implicit interpolation alignment unit to obtain a second fusion alignment feature; The first spatiotemporal fusion feature and the extracted feature are concatenated and input into the third residual convolution unit to obtain a second intermediate feature; The second intermediate feature and the second fused alignment feature are input into the second spatiotemporal feature fusion unit to obtain the second spatiotemporal fusion feature.

7. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 6, characterized in that: The second implicit interpolation alignment unit includes: a third implicit interpolation alignment sub-unit, a fourth implicit interpolation alignment sub-unit and a second deformable alignment unit; Obtaining a previous frame of spatiotemporal fusion features based on a previous frame of video frame image of the current video frame image after being processed by the forward propagation module; The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous video frame image, and the spatiotemporal fusion feature of the previous frame are input into the third implicit interpolation alignment subunit to obtain a third alignment feature; Based on the two video frame images above the current video frame image, the two frames of spatiotemporal fusion features are obtained after being processed by the forward propagation module; The first spatiotemporal fusion feature, the second optical flow information between the current video frame image and the previous two video frame images, and the spatiotemporal fusion features of the previous two frames are input into the fourth implicit interpolation alignment subunit to obtain a fourth alignment feature; The third alignment feature and the fourth alignment feature are input into the second deformable alignment unit to obtain the second fused alignment feature.

8. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 4, characterized in that: The first alignment feature acquisition method is: Obtaining a first pixel displacement in the x and y directions between a current video frame image and a next video frame image based on the first optical flow information; Obtaining a next frame window area based on the first pixel displacement and the next frame spatiotemporal fusion features; Obtaining a first query pixel based on the extracted features; obtaining a first coordinate code of the first query pixel based on the first pixel displacement; Obtaining a first Q value based on the first query pixel and the first coordinate code input into a query information coordinate network; Obtaining a second coordinate code of the next frame window area based on the next frame window area; Based on the next frame window area and the second coordinate code, they are input into the key information coordinate network and the value information coordinate network respectively, and a first K value and a first V value are obtained accordingly; Obtaining an alignment result for each pixel in the extracted feature based on the first Q value, the first K value, and the first V value; The first alignment feature is obtained based on the alignment results of all pixels.

9. The video frame reconstruction method based on implicit interpolation alignment and feature fusion according to claim 4, characterized in that: The reconstruction module includes: a fourth residual convolution unit, a pixel shuffling unit and a third upsampling unit; The second spatiotemporal fusion unit is sequentially input into the fourth residual convolution unit and the pixel shuffling unit to obtain a high-resolution frame image; The current frame image is input to the third upsampling unit for processing and then fused with the high-resolution frame image to obtain the high-resolution reconstructed video frame.

10. A video frame reconstruction system based on implicit interpolation alignment and feature fusion, configured to execute the video frame reconstruction method based on implicit interpolation alignment and feature fusion according to any one of claims 1 to 9, characterized in that: include: Data acquisition module, first feature extraction module, second feature extraction module, third feature extraction module and reconstruction output module; The data acquisition module is used to acquire a plurality of consecutive video frame images to be processed and input them into corresponding video frame reconstruction branches; The first feature extraction module is configured to obtain extracted features based on the current video frame image input to the feature extraction module; The second feature extraction module is configured to input the extracted features and the first optical flow information between the current video frame image and the next two frames into the backward propagation module to obtain a first spatiotemporal fusion feature; The third feature extraction module is configured to input the first spatiotemporal fusion feature, the extracted feature, and the second optical flow information between the current video frame image and the previous two frames into the forward propagation module to obtain a second spatiotemporal fusion feature; The reconstruction output module is configured to input the second spatiotemporal fusion feature and the current video frame image into a reconstruction module to obtain a high-resolution reconstructed video frame corresponding to the current video frame image; Repeat the above process to obtain the high-resolution reconstructed video frames corresponding to all the video frame images.

Images