Video frame insertion reconstruction method and system based on spatio-temporal feature fusion

Abstract

The invention discloses a video frame insertion reconstruction method and system based on spatio-temporal feature fusion. The method comprises the following steps: dividing a video data set into a training set and a test set; feature recovery is carried out on the complex regions in the time dimension and the space dimension to obtain reconstructed features; estimating a plurality of fine-grained optical streams through cascaded multi-stream estimation units in a coarse-to-fine strategy; guiding a pixel warping process through a plurality of attention weight maps of the optical flow warping unit; weighting and mixing all the predicted distortion frames through the mixed mask to generate a reconstructed target frame; and carrying out hierarchical stacking on the multi-stream estimation unit, the attention weight map and the mixed mask to obtain a spatio-temporal feature fused video frame insertion reconstruction model. According to the method, modeling is carried out on target features through two layers of time and space, so that fine textures of the regions are recovered, and the processing capability of the model on complex textures is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video frame insertion and reconstruction method and system based on spatiotemporal feature fusion. Background Art

[0002] With the rapid development of multimedia and communication technologies, video, with its vivid and intuitive nature, has become the most important means of information transmission after text and images. In recent years, the video-related industry has rapidly developed, with video conferencing and online teaching in work and school, as well as live broadcasts and short videos in daily life and entertainment, becoming an indispensable way to improve team collaboration efficiency and enrich people's cultural and entertainment lives.

[0003] Videos are composed of multiple frames. Resolution refers to the number of pixels in each frame; higher resolutions represent clearer and more detailed video images. Frame rate, on the other hand, refers to the number of frames displayed per second; higher frame rates represent smoother and more coherent video images. Generally speaking, a video with a frame rate of 30 frames per second is considered coherent and acceptable. When the frame rate reaches 60 frames per second, the visual quality is significantly improved and is considered a high refresh rate video. Therefore, video interpolation and reconstruction, as a technology for converting low-frame-rate video to high-frame-rate video, has gained widespread attention.

[0004] Traditional video interpolation and reconstruction methods mainly calculate video frames at corresponding time domain positions by establishing mathematical statistical models or numerical models based on physical equations. This method based on mathematical modeling has strong interpretability, but due to the complexity of the model, it often requires high computing resources and long calculation time. In addition, because the assumptions relied on by the model deviate from the real scene, it is easy to make mistakes when calculating pixel motion when processing videos with complex motion scenes, and the predicted frames often have obvious distortion or artifacts. With the continuous development of artificial intelligence, video frame prediction methods based on deep learning have gradually become mainstream. Compared with traditional methods, deep learning-based methods can provide more accurate video picture content. Moreover, as the amount of video data continues to grow, data-driven training methods make deep learning-based methods more universal and better able to handle motion in complex scenes.

[0005] Currently, mainstream deep learning methods generally consist of two stages. In the first stage, convolutional neural networks (CNNs) or Transformer networks are typically used to extract motion information from previous and subsequent frames. However, CNN-based methods are often limited by the fixed convolution kernel size, making them unable to perceive motion beyond the limited receptive field, resulting in blurry interpolated results or artifacts. Transformer-based methods can capture dependencies between long-range pixels, overcoming the limited receptive field of fixed convolution kernels. However, they are less capable of learning local texture details than convolution operations, leading to overly smoothed interpolated results. Furthermore, existing Transformer-based methods typically implement self-attention within a fixed-size temporal window, resulting in severe artifacts in the interpolated results when facing large inter-frame motion. The second stage uses motion information to estimate optical flow or convolution kernel sampling. Optical flow-based methods typically only utilize pixels at a single location in the input frame, resulting in a lack of effective information in the interpolated results. Convolution kernel-based methods have undergone a series of evolutionary stages, with the current state-of-the-art methods adaptively offsetting each sampling position within the convolution kernel, significantly increasing the degrees of freedom in sampling positions. However, the number of sampling points is usually set to 3×3 or 5×5, which shows a quadratic growth law. The number of sampling points lacks freedom, and too many sampling points may cause negative effects. Summary of the Invention

[0006] The embodiments of the present invention provide a video interpolation reconstruction method and system based on spatiotemporal feature fusion, which is used to solve the problem in the prior art that it is impossible to reconstruct objects with large-scale motion characteristics in the video with high quality, resulting in the reconstructed interpolated video having artifacts and noise.

[0007] On the one hand, an embodiment of the present invention provides a video frame insertion and reconstruction method based on spatiotemporal feature fusion, comprising:

[0008] Divide the video dataset into training set and test set;

[0009] Perform feature recovery on complex regions in both time and space dimensions to obtain reconstructed features;

[0010] Estimate multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-flow estimation units;

[0011] The pixel warping process is guided by multiple attention weight maps of the optical flow warping unit;

[0012] The reconstructed target frame is generated by weighted mixing of all predicted warped frames through the mixing mask;

[0013] The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked to obtain a video interpolation reconstruction model with spatiotemporal feature fusion;

[0014] The video frame insertion reconstruction model is trained using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model;

[0015] The spatiotemporal feature fusion video interpolation reconstruction model is used to perform video interpolation reconstruction on the test set to obtain restored target frame data.

[0016] In a possible implementation, the performing feature recovery on a complex region in both time and space dimensions to obtain reconstructed features includes:

[0017] After calculating self-attention in time and space respectively, the attention intensity matrix of the two levels of time and space regions is obtained within the frame and between frames;

[0018] Similar fine features are extracted from the attention intensity matrix described in the two levels of time and space to obtain reconstructed features.

[0019] In a possible implementation, estimating multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-stream estimation units includes:

[0020] The motion offsets of multiple optical flow fields and the attention weight maps corresponding to the optical flow fields in the horizontal and vertical directions are predicted through cascaded multi-stream estimation units.

[0021] In one possible implementation, guiding the pixel warping process through multiple attention weight maps of the optical flow warping unit includes:

[0022] predicting a plurality of said optical flow fields, associating each position in the features of the output image with features at a plurality of positions in the features of the input image;

[0023] Integrating the features of the output image with the features of the input image through the attention weight map;

[0024] Pixel warping of the integrated features of the input image is performed using the optical flow field and the attention weight map.

[0025] In a possible implementation, the generating of the reconstructed target frame by weighted mixing all predicted warped frames through a mixing mask includes:

[0026] The mixing mask for each consecutive time frame is predicted by CNN.

[0027] In a possible implementation, the multi-stream estimation unit, the attention weight map, and the hybrid mask are hierarchically stacked to obtain a video interpolation reconstruction model with spatiotemporal feature fusion, including:

[0028] The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked through a U-shaped structure as the overall network structure to obtain the video interpolation reconstruction model with spatiotemporal feature fusion.

[0029] On the other hand, an embodiment of the present invention provides a video frame insertion and reconstruction system based on spatiotemporal feature fusion, comprising:

[0030] Video data partitioning module, used to divide the video data set into training set and test set;

[0031] The spatiotemporal feature fusion module is used to recover features of complex areas in both time and space dimensions to obtain reconstructed features;

[0032] Multi-stream refinement module, which estimates multiple fine-grained optical flows in a coarse-to-fine strategy through cascaded multi-stream estimation units, and guides the pixel warping process through multiple attention weight maps of the optical flow warping unit;

[0033] A multi-stream guided fusion module is used to generate a reconstructed target frame by weighted blending of all predicted warped frames through a blending mask;

[0034] A model construction module, configured to hierarchically stack the multi-stream estimation unit, the attention weight map, and the hybrid mask to obtain a video interpolation reconstruction model with spatiotemporal feature fusion;

[0035] A model training module, configured to train the video frame insertion reconstruction model using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model;

[0036] The video frame interpolation module is used to use the spatiotemporal feature fusion video frame interpolation reconstruction model to perform video frame interpolation reconstruction on the test set to obtain restored target frame data.

[0037] The video frame insertion and reconstruction method and system based on spatiotemporal feature fusion in the present invention have the following advantages:

[0038] (1) By modeling the target features at both the temporal and spatial levels, the fine textures of these areas are restored, thus enhancing the model's ability to process complex textures.

[0039] (2) Through the multi-stream refinement module, the target motion optical flow is estimated at different levels, which enhances the motion estimation capability of multiple targets in complex motion scenes.

[0040] (3) Through the multi-stream guided fusion module, the features of multiple consecutive frames are effectively fused, the noise and error of the reconstructed features are eliminated, the better high-frequency information is restored, and the generalization ability of the model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a video frame insertion and reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0043] Figure 2 A model architecture diagram of a video frame insertion and reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the spatiotemporal feature fusion module architecture of a video frame insertion reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of intra-frame cross attention of a spatiotemporal feature fusion module of a video interpolation reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of an inter-frame expansion attention of a spatiotemporal feature fusion module of a video interpolation reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0047] Figure 6 A multi-stream refinement module for a video frame insertion and reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0048] Figure 7 A multi-stream guided fusion module for a video frame insertion reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0049] Figure 8 A comparison diagram of the results of an embodiment of a video frame insertion and reconstruction system based on spatiotemporal feature fusion provided by an embodiment of the present invention;

[0050] Figure 9 This is a structural block diagram of a video frame insertion and reconstruction system based on spatiotemporal feature fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Figure 1 A flowchart of a video frame insertion and reconstruction method based on spatiotemporal feature fusion provided by an embodiment of the present invention; an embodiment of the present invention provides a video frame insertion and reconstruction method based on spatiotemporal feature fusion, comprising:

[0053] Divide the video dataset into training set and test set;

[0054] Perform feature recovery on complex regions in both time and space dimensions to obtain reconstructed features;

[0055] Estimate multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-flow estimation units;

[0056] The pixel warping process is guided by multiple attention weight maps of the optical flow warping unit;

[0057] The reconstructed target frame is generated by weighted mixing of all predicted warped frames through the mixing mask;

[0058] The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked to obtain a video interpolation reconstruction model with spatiotemporal feature fusion;

[0059] The video frame insertion reconstruction model is trained using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model;

[0060] Using the spatiotemporal feature fusion video interpolation reconstruction model to perform video interpolation reconstruction on the test set to obtain restored target frame data;

[0061] The reconstructed features obtained by performing feature recovery on the complex region in both time and space dimensions include:

[0062] After calculating self-attention in time and space respectively, the attention intensity matrix of the two levels of time and space regions is obtained within the frame and between frames;

[0063] Extract similar fine features from the attention intensity matrix described in the two levels of time and space to obtain reconstructed features;

[0064] The method of estimating multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-flow estimation units includes:

[0065] Predicting the motion offsets of multiple optical flow fields and the attention weight maps corresponding to the optical flow fields in the horizontal and vertical directions through a cascaded multi-stream estimation unit;

[0066] The pixel warping process guided by multiple attention weight maps of the optical flow warping unit includes:

[0067] predicting a plurality of said optical flow fields, associating each position in the features of the output image with features at a plurality of positions in the features of the input image;

[0068] Integrating the features of the output image with the features of the input image through the attention weight map;

[0069] Perform pixel warping of the integrated features of the input image using the optical flow field and the attention weight map;

[0070] The reconstructed target frame is generated by weighted mixing of all predicted warped frames through a mixed mask, comprising:

[0071] Predicting the mixing mask for each consecutive time frame through CNN;

[0072] The video frame insertion reconstruction model obtained by layering the multi-stream estimation unit, the attention weight map and the hybrid mask to obtain spatiotemporal feature fusion includes:

[0073] The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked through a U-shaped structure as the overall network structure to obtain the video interpolation reconstruction model with spatiotemporal feature fusion.

[0074] For example, Figure 3 、 4 As shown in Figure 5, a spatiotemporal feature fusion module is designed. Regions with similar complex textures are searched from various texture contents, and a texture-aware restoration effect is achieved by performing fine self-attention calculations on these regions. The spatiotemporal feature fusion module consists of a cross-attention branch for intra-frame spatial features and an expanded attention branch for inter-frame temporal features. The intra-frame spatial feature cross-attention branch is divided into a CNN branch and a Transformer branch. The calculation formula of the CNN branch is:

[0075]

[0076] Where: Denotes the features of the cross-attention branch of the spatial features within the input frame, DWConv denotes the depthwise convolution with a kernel size of 3×3, LN denotes layer normalization, It is the output feature after being processed by the CNN branch.

[0077] Perform cross-modulation between CNN and Transformer branches, and use Sigmoid to generate attention weights S in the channel dimension. c , and applied to the value matrix V of the Transformer branch:

[0078]

[0079]

[0080] Where: GAP represents global average pooling, δ represents activation function, σ represents Sigmoid operation, W1 and W2 represent 1×1 convolution, Represents element-wise multiplication.

[0081] In the Transformer branch, the features Divide into non-overlapping sub-windows of size M × M. Taking one of the windows X as an example, its query matrix Q, key matrix K and value matrix V are calculated as:

[0082] Q=XW Q ,K=XW K ,V=XW V

[0083] Where: W Q 、W K and W V is the projection matrix shared across different windows,

[0084] The attention matrix that enhances high-frequency context details is calculated by the query matrix Q and the key matrix K. The calculation formula is:

[0085]

[0086]

[0087] Att hp =Att-Att lp

[0088] Att′=Att lp +(1+β)Att hp

[0089] Where: All-one matrix M 2 is a fixed number of pixels in the sub-window. And the complementary part of the low-pass filter is used as a high-frequency filter; where Att is the attention matrix for query matrix Q and key matrix K to enhance high-frequency context details; Att lp Indicates a low-pass filter where every element in the matrix is ​​equal (equivalent to the DC component); Att hpThis is equivalent to subtracting a low-pass filter from Att, that is, removing the low-pass part or DC component in Att and leaving the high-frequency part Att. hp

[0090] Use the new attention weight Att′ to multiply the corresponding value matrix V to obtain the output features of the attention branch

[0091]

[0092] Perform cross-modulation between the Transformer and CNN branches, and use Sigmoid to generate spatial attention weights S s , and applied to the depthwise convolution output

[0093]

[0094]

[0095] The CNN and Transformer branch features are cascaded as the output of the intra-frame spatial feature cross attention branch, where W3 and W4 are weight matrices, and W3 is the weight matrix. Perform linear transformation, W4 is The result is linearly transformed.

[0096] For the inter-frame temporal feature expansion attention branch, feature extraction is performed using the Transformer-based expansion attention interaction method. Given a pair of video frame features F i and F j , where i,j∈[0,1,…,T-1] and i≠j, the frame feature F i Input into the linear layer to get the query matrix Q i , similarly, the frame feature F j Input into two linear layers respectively to obtain the key matrix K j Sum matrix V j In K j When sampling, the sampling position is expanded outward with the expansion rate δ=|ji|, that is, the sampling position becomes (p o +δp ξ ). In order to adaptively mix K j Middle K 2 The spatial information of the sampling position is first calculated using the Softmax function i (p o ) and K j The attention weight of the feature vector at the sampling point in:

[0097]

[0098] in,

[0099] W ξ is the attention weight of the ξ-th sampling position. Q i is the query matrix. K j is the bond matrix. p o is the original position, δ=|ji| is the expansion rate, p ξ is the offset of the ξ-th sampling position. 2 is the total number of sampling locations

[0100] Then, using the attention weight and feature vector K j (p o +δp ξ ), the corresponding fused key vector can be obtained:

[0101]

[0102] In the same way, the spatial fusion value vector V can be calculated j→i (p o ).

[0103] Finally, the spatial features of other frames are further adaptively aggregated along the time dimension. Specifically, Q is first calculated. i (p o ) and K j→i (p o ) between attention weights:

[0104]

[0105] Next, we further use the attention weights to fuse the temporal context and obtain the output of the inter-frame temporal feature expansion attention branch:

[0106]

[0107] like Figure 6 As shown, a multi-stream motion estimation unit is designed to predict multiple optical flow fields and its corresponding attention map Where K refers to the number of optical flow fields, which have a channel dimension of 2 and include both horizontal and vertical offsets. A small CNN with three layers of 2D convolution is used to predict multiple optical flow fields and their corresponding attention maps, which include both horizontal and vertical motion offsets. Multiple flow fields and attention maps are generated from different channels of the convolutional network output feature map. Specifically, multiple flows correspond to the first 2K channels of the feature map, and the attention map corresponds to the last K channels.

[0108] The optical flow warping unit is designed to associate each position in the output image or feature with features from multiple positions in the input image or feature by predicting multiple optical flow fields, and effectively integrate them together using the attention maps corresponding to the optical flow fields. Each input frame is warped using the refined multiple flow fields predicted at the highest resolution and the final attention map. The process expression is described as:

[0109]

[0110] Where: p refers to the pixel position, p+o(k,p) is the kth sampling position, and a(k,p) is the attention weight corresponding to the sampling position. In addition, for non-integer grid positions, their values ​​are obtained by bilinear interpolation, I: represents the input image or feature (Input), I(p+o(k,p)) represents the image / feature value at position p+o(k,p). o: represents the offset (Offset) of the optical flow field, o(k,p) is the offset of the kth sampling position relative to p. a: represents the attention weight (Attentionweights), a(k,p) is the attention weight of the kth sampling position at p.

[0111] The core of this stage is to use multi-scale features {F s ,s=0,1,2}, where s=0,1,2 represent different scales from low resolution to high resolution. Specifically, given the feature First, use the multi-flow field predicted by the previous layer and attention map To twist it:

[0112]

[0113] Where: U represents the bilinear upsampling operation. Then the distorted features are Input to the multi-stream motion estimation unit to estimate the initial residual multi-flow field and attention map Finally, use and Further distorting features And input it into another modulo multi-flow motion estimation unit to estimate the refined multi-flow field And the final attention map For scale s, its corresponding multi-flow field is expressed as follows:

[0114]

[0115] Each input frame is warped using the refined multi-flow field predicted at the highest resolution and the final attention map:

[0116]

[0117] Where: is the distorted input frame, I t is the original input frame;

[0118] like Figure 7 As shown in the figure, a multi-stream guided fusion module is designed to use 2D CNN to obtain the mixed mask predicted by each continuous time frame. Used to blend all warp frames

[0119] The process expression is:

[0120]

[0121] Where: ⊙ represents element-by-element multiplication, is the intermediate frame of the final output;

[0122] The U-shaped structure is used as the overall network structure. The encoder is composed of three stacked spatiotemporal feature fusion modules. The encoder and decoder are connected by a spatiotemporal feature fusion module. The decoder splices the input feature map and the encoder features of the corresponding scale along the channel dimension and outputs multi-level refined features to the multi-stream refinement module.

[0123] By using the multi-stream guided fusion module, the mixed information of multiple consecutive frames is effectively utilized to jointly predict the video frame to be interpolated, avoiding the influence of single frame prediction error on the reconstruction of the interpolated frame, thereby improving the generalization ability of the model.

[0124] Multiple spatiotemporal feature fusion modules are stacked in a U-shaped structure and connected with the multi-stream refinement module in layers, and finally input into the multi-stream guided fusion module. Figure 2 As shown, Figure 2A structural diagram of video interpolation reconstruction based on spatiotemporal feature fusion is provided for an embodiment of the present invention. The network uses a 3×3 convolution and a LeakyReLU activation function as shallow feature extraction, and then the shallow features are input into the encoder. The encoder is composed of a stack of 3D convolution layers with a stride of (1,2,2) and a convolution kernel of (3,3,3) for spatial downsampling, and a spatiotemporal feature fusion module, and a spatiotemporal feature fusion module is connected between the encoder and the decoder. A single 3D transposed convolution layer with a stride of (1,2,2) and a convolution kernel of (3,3,3) is used in the decoder to upsample the feature map. The upsampled feature map and the encoder features of the corresponding scale are spliced ​​along the channel dimension and then input into the next decoder. In the multi-stream refinement module, multiple fine-grained flow fields and corresponding attention maps are predicted in a coarse-to-fine manner through cascaded multi-stream motion estimation units. Specifically, a multi-stream motion estimation unit first estimates multiple coarse optical flow fields and corresponding attention maps. The optical flow warping unit then performs backward warping to obtain multiple warped features. These features are then combined using attention weights and fed into the next multi-stream motion estimation unit to predict the residual optical flow, thereby obtaining multiple refined optical flow fields. This process continues until the highest resolution scale is reached. This progressively refined multi-stream approach can better handle complex motion scenes. Finally, in the multi-stream guided fusion module, the hybrid mask predicted by the highest resolution features is combined with the final refined warped input frame guided by multiple optical flow fields to generate an interpolated frame output.

[0125] According to the gradient descent method based on Adam optimizer, the video data training set is used to train the video interpolation model based on spatiotemporal feature fusion.

[0126] The trained video interpolation model based on spatiotemporal feature fusion is used to perform super-resolution reconstruction on the video test set to obtain the reconstructed interpolated frames.

[0127] In a possible embodiment, tests were conducted on two mainstream video frame reconstruction datasets (DAVIS dataset and Vimeo-90K dataset), as shown in Table 1: PSNR and SSIM numerical comparison table on two video interpolation datasets, DAVIS and Vimeo-90K. The present invention compares the PSNR and SSIM numerical values ​​with other benchmark methods on two video interpolation datasets, DAVIS and Vimeo-90K. The compared benchmark methods include AdaCoF, FLAVR and VFIT. It can be seen from the quantitative results in Table 1 that the numerical indicators of the video interpolation model based on spatiotemporal feature fusion (Spatial-Temporal FusionNetwork, STFNet) of the present invention are significantly higher than those of other methods.

[0128] Table 1 Comparison of PSNR and SSIM values ​​on two video interpolation datasets: DAVIS and Vimeo-90K

[0129]

[0130] Figure 8 The figure shows some interpolated video frames reconstructed by the above-mentioned baseline method and the video interpolation model STFNet based on spatiotemporal feature fusion proposed in this invention (where Overlay represents the overlap of the previous and next frame inputs, which can reflect the speed of motion and occlusion. Next are the predicted intermediate frames obtained by each video frame interpolation method, as well as the real target frame, where target is the target frame). Figure 7 It can be seen that the intermediate frames reconstructed by the video interpolation model STFNet based on spatiotemporal feature fusion proposed in this invention can better handle large-scale target motion, produce fewer artifacts, and generate better detailed texture information.

[0131] Figure 9 This is a structural block diagram of a video frame insertion and reconstruction system based on spatiotemporal feature fusion provided by an embodiment of the present invention. The embodiment of the present invention provides a video frame insertion and reconstruction system based on spatiotemporal feature fusion, including:

[0132] Video data partitioning module, used to divide the video data set into training set and test set;

[0133] The spatiotemporal feature fusion module is used to recover features of complex areas in both time and space dimensions to obtain reconstructed features;

[0134] Multi-stream refinement module, which estimates multiple fine-grained optical flows in a coarse-to-fine strategy through cascaded multi-stream estimation units, and guides the pixel warping process through multiple attention weight maps of the optical flow warping unit;

[0135] A multi-stream guided fusion module is used to generate a reconstructed target frame by weighted blending of all predicted warped frames through a blending mask;

[0136] A model construction module, configured to hierarchically stack the multi-stream estimation unit, the attention weight map, and the hybrid mask to obtain a video interpolation reconstruction model with spatiotemporal feature fusion;

[0137] A model training module, configured to train the video frame insertion reconstruction model using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model;

[0138] The video frame interpolation module is used to use the spatiotemporal feature fusion video frame interpolation reconstruction model to perform video frame interpolation reconstruction on the test set to obtain restored target frame data.

[0139] Exemplarily, the video data partitioning module divides the video dataset into a training set and a test set. The spatiotemporal feature fusion module performs self-attention calculations on both time and space, obtains attention strength matrices for both intra-frame and inter-frame spatiotemporal regions, and extracts similar fine features for interpolation reconstruction. The multi-stream refinement module, composed of alternating multi-stream motion estimation units and optical flow warping units, fully predicts inter-frame motion information and estimates the motion positions of consecutive frames based on optical flow. The multi-stream guided fusion module utilizes feature refinement information from the spatiotemporal feature fusion module at the corresponding scale and motion information predicted by the multi-stream refinement module based on optical flow variations to obtain predicted frames with rich frequency domain information. The model construction module stacks the above modules in a U-shaped structure, stacking the spatiotemporal feature fusion module and the multi-stream refinement module to obtain a multi-scale spatiotemporal feature fusion video interpolation reconstruction model. The model training module trains the training set using a gradient descent method based on the Adam optimizer to obtain a trained spatiotemporal feature fusion video interpolation reconstruction model. The video interpolation value module then uses the trained spatiotemporal feature fusion video interpolation reconstruction model to reconstruct the test set, obtaining reconstructed video interpolation images. Through the spatiotemporal feature fusion module, target features can be modeled at both the temporal and spatial levels, allowing these areas to restore finer textures and enhancing the model's ability to process complex textures. Through the multi-stream refinement module, target motion optical flows are estimated at different levels, enhancing the motion estimation capability of multiple targets in complex motion scenes. In addition, the module also effectively fuses features of multiple consecutive frames through the multi-stream guided fusion module, eliminating noise and errors in the reconstructed features, restoring better high-frequency information, and improving the generalization capability of the model.

[0140] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications that fall within the scope of the present invention and the preferred embodiments.

[0141] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A video frame insertion and reconstruction method based on spatiotemporal feature fusion, characterized in that: include: Divide the video dataset into training set and test set; Perform feature recovery on complex regions in both time and space dimensions to obtain reconstructed features; Estimate multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-flow estimation units; The pixel warping process is guided by multiple attention weight maps of the optical flow warping unit; The reconstructed target frame is generated by weighted mixing of all predicted warped frames through the mixing mask; The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked to obtain a video interpolation reconstruction model with spatiotemporal feature fusion; The video frame insertion reconstruction model is trained using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model; The spatiotemporal feature fusion video interpolation reconstruction model is used to perform video interpolation reconstruction on the test set to obtain restored target frame data.

2. The video frame insertion and reconstruction method based on spatiotemporal feature fusion according to claim 1, characterized in that: The reconstructed features obtained by performing feature recovery on the complex region in both time and space dimensions include: After calculating self-attention in time and space respectively, the attention intensity matrix of the two levels of time and space regions is obtained within the frame and between frames; Similar fine features are extracted from the attention intensity matrix described in the two levels of time and space to obtain reconstructed features.

3. The video frame insertion and reconstruction method based on spatiotemporal feature fusion according to claim 1, characterized in that: The method of estimating multiple fine-grained optical flows using a coarse-to-fine strategy through cascaded multi-flow estimation units includes: The motion offsets of multiple optical flow fields and the attention weight maps corresponding to the optical flow fields in the horizontal and vertical directions are predicted through cascaded multi-stream estimation units.

4. The video frame insertion and reconstruction method based on spatiotemporal feature fusion according to claim 3, characterized in that: The pixel warping process guided by multiple attention weight maps of the optical flow warping unit includes: predicting a plurality of said optical flow fields, associating each position in the features of the output image with features at a plurality of positions in the features of the input image; Integrating the features of the output image with the features of the input image through the attention weight map; Pixel warping of the integrated features of the input image is performed using the optical flow field and the attention weight map.

5. The video frame insertion and reconstruction method based on spatiotemporal feature fusion according to claim 1, characterized in that: The reconstructed target frame is generated by weighted mixing of all predicted warped frames through a mixed mask, comprising: The mixing mask for each consecutive time frame is predicted by CNN.

6. The video frame insertion and reconstruction method based on spatiotemporal feature fusion according to claim 1, characterized in that: The video frame insertion reconstruction model obtained by layering the multi-stream estimation unit, the attention weight map and the hybrid mask to obtain spatiotemporal feature fusion includes: The multi-stream estimation unit, the attention weight map and the hybrid mask are hierarchically stacked through a U-shaped structure as the overall network structure to obtain the video interpolation reconstruction model with spatiotemporal feature fusion.

7. A video frame insertion and reconstruction system based on spatiotemporal feature fusion, characterized in that: include: Video data partitioning module, used to divide the video data set into training set and test set; The spatiotemporal feature fusion module is used to recover features of complex areas in both time and space dimensions to obtain reconstructed features; Multi-stream refinement module, which estimates multiple fine-grained optical flows in a coarse-to-fine strategy through cascaded multi-stream estimation units, and guides the pixel warping process through multiple attention weight maps of the optical flow warping unit; A multi-stream guided fusion module is used to generate a reconstructed target frame by weighted blending of all predicted warped frames through a blending mask; A model construction module, configured to hierarchically stack the multi-stream estimation unit, the attention weight map, and the hybrid mask to obtain a video interpolation reconstruction model with spatiotemporal feature fusion; A model training module, configured to train the video frame insertion reconstruction model using the training set to obtain a spatiotemporal feature fusion video frame insertion reconstruction model; The video frame interpolation module is used to use the spatiotemporal feature fusion video frame interpolation reconstruction model to perform video frame interpolation reconstruction on the test set to obtain restored target frame data.

Images