A training method for high-resolution video enhancement models based on continuous motion

Abstract

The present invention discloses a training method for a high-resolution video enhancement model based on continuous motion and a video interpolation method. The former comprises: constructing a video interpolation model, including a feature extractor, a motion prior extractor, an optical flow estimator, and a correction network; based on a first video data set, pre-training the model is performed using a training strategy with random interpolation time and a preset multiple loss function; the multiple loss function is constructed based on a Laplace loss function and an optical flow loss function; based on a second video data set containing long-term high-resolution video clips, model optimization training is performed again to obtain a trained video interpolation model; the present invention effectively eliminates the motion uncertainty between different frames by modeling continuous motion between frames, thereby realizing accurate motion estimation of specific positions between frames by the model, which can improve the accuracy and effect of high-resolution video interpolation, especially when processing large-scale motion and complex scenes, with stronger robustness and accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and image processing technology, and in particular relates to a training method for a high-resolution video enhancement model based on continuous motion. Background Art

[0002] Video Frame Interpolation (VFI) is an important video processing technology that aims to generate missing intermediate frames between existing video frames, thereby improving the temporal resolution of the video and enhancing the viewing experience. Video interpolation can not only be used for general video enhancement, but is also widely used in slow motion generation, video compression, image restoration, and animation production. In recent years, with the rapid development of deep learning technology, video interpolation methods have been continuously innovated, especially in the application of high-resolution videos and large-scale motion scenes. However, although the existing technology has good performance in theory, it still faces many challenges in practical applications, especially in high-resolution videos, large-scale motion and complex scenes, the interpolation effect is far from what expected.

[0003] In recent years, optical flow-based interpolation methods have been widely used, particularly when dealing with small motion or stable video content. For example, in the paper "A unified pyramid recurrent network for video frame interpolation," Xin Jin et al. proposed a pyramid-based recurrent network method that generates intermediate frames through optical flow estimation. This method guides the interpolation process by calculating optical flow using the correlation between adjacent frames. However, the main drawback of this method is that the calculation of optical flow relies on correlation matching and lacks sufficient interpretability. In particular, when dealing with high-resolution videos and large-scale motion, it often fails to effectively capture continuous motion trajectories. Due to the instability of motion and errors in optical flow estimation, the generated intermediate frames appear blurred and distorted.

[0004] To address the problem of frame interpolation in scenes with large motion, Chunxu Liu et al. proposed a sparse global matching method in "Sparse global matching for video frame interpolation with large motion." This method attempts to enhance its ability to handle large motion and high-resolution video interpolation by introducing additional datasets. This method extracts global motion information through sparse matching and uses this global information to synthesize intermediate frames. However, while this method theoretically provides an effective strategy for handling scenes with large motion, several practical issues remain. First, while this method introduces a new dataset during training, its training methods and strategies lack sufficient systematic exploration, resulting in suboptimal performance for high-resolution video interpolation. Second, its global attention mechanism results in computational complexity that increases quadratically with video resolution, significantly increasing computational requirements when processing high-resolution videos. Furthermore, in scenes with dramatic dynamics, the locality limitations of sparse matching methods limit their performance in complex scenes, making them unable to fully meet the requirements of high-resolution and large-scale motion interpolation.

[0005] In addition, Guozhen Zhang et al. proposed a state-space model-based interpolation method in "VFIMamba: Video Frame Interpolation with State Space Models", which mainly explored the training method of fixed-time interpolation. By using the state-space model, the model can effectively transfer information between frames to generate accurate intermediate frames. Although this method performs well in fixed interpolation scenarios, its insufficient exploration of multi-frame insertion (i.e., generating multiple intermediate frames) limits its application in complex dynamic scenes. For example, in high-speed motion and rapidly changing scenes, the fixed interpolation method finds it difficult to capture the details of continuous motion, resulting in distortion and blurring of the generated intermediate frames.

[0006] A key challenge facing high-resolution video interpolation is maintaining accuracy and efficiency without introducing excessive computational overhead. Many existing methods for high-resolution video interpolation often require extensive computational resources and complex model structures. When processing scenes with large motion, accurately modeling motion trajectories while maintaining detail and clarity in the interpolated images remains a major bottleneck. High-resolution video processing requires a higher level of detail preservation, but existing methods often neglect how to efficiently handle complex motion at high resolutions, resulting in frequent distortions in detail and texture preservation.

[0007] While existing video interpolation methods have achieved promising results under certain conditions, they generally suffer from the following issues: First, accurately modeling motion trajectories in complex scenes, especially in situations involving high resolution and large-scale motion, remains an unsolved problem. Second, existing methods mostly focus on a specific type of scene or task and lack broad adaptability to a wide range of scenarios and tasks. Especially when faced with videos with dramatic dynamic changes, existing interpolation methods often fail to accurately capture motion details, resulting in blurry and distorted generated images. Finally, how to effectively conduct efficient training and reduce computational overhead remains a key research focus. Summary of the Invention

[0008] To address the above-mentioned problems in the prior art, the present invention provides a training method for a high-resolution video enhancement model based on continuous motion and a video frame insertion method. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0009] In a first aspect, an embodiment of the present invention provides a method for training a high-resolution video enhancement model based on continuous motion, the method comprising:

[0010] Constructing a video interpolation model; wherein the video interpolation model includes a feature extractor, a motion prior extractor, an optical flow estimator, and a correction network; the feature extractor is used to extract multi-dimensional features from two input frames; the motion prior extractor is used to obtain displacement information of the intermediate frame based on the high-dimensional features, a correlation measure, and the interpolation time; the optical flow estimator is used to obtain optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining a preliminary intermediate frame image; the correction network is used to correct the preliminary intermediate frame image based on the multi-dimensional features and output a predicted intermediate frame image;

[0011] Based on the first video dataset, the video interpolation model is pre-trained using a training strategy with random interpolation time and a preset multiple loss function; wherein the multiple loss function is constructed based on a Laplace loss function and an optical flow loss function;

[0012] Based on a second video dataset containing long high-resolution video clips, the pre-trained video interpolation model is optimized and trained to obtain a trained video interpolation model; the trained video interpolation model is used to output an interpolated intermediate frame image for two input frames.

[0013] In one embodiment of the present invention, the feature extractor includes three CNN modules and two Transformer modules; wherein the two Transformer modules are used to output high-dimensional features.

[0014] In one embodiment of the present invention, the process of the motion prior extractor obtaining displacement information of the intermediate frame based on the high-dimensional features output by the feature extractor and the correlation metric and the interpolation time includes:

[0015] Using the two motion prior extraction modules of the motion prior extractor, high-dimensional features of two frames of images are received; wherein the two frames of images are a first frame of image and a second frame of image;

[0016] For each pixel position, calculating a correlation value between the pixel value of the pixel position in the first frame of image and the pixel values of each neighboring region of the pixel position in the second frame of image based on the high-dimensional features of the two frames of image;

[0017] The correlation value obtained at each pixel position is processed by Softmax;

[0018] According to the correlation value after Softmax processing and the coordinates of each pixel position, the displacement information of each pixel is obtained;

[0019] The displacement information of the corresponding pixel in the intermediate frame is obtained according to the product of the displacement information of each pixel and the interpolation time.

[0020] In one embodiment of the present invention, obtaining the displacement information of each pixel based on the correlation value after Softmax processing and the coordinates of each pixel position includes:

[0021] Perform tensor transformation on the correlation value after Softmax processing;

[0022] For each pixel position, after mapping its coordinates from two dimensions to the same dimension as the feature, the current correlation value of the pixel position is multiplied by its coordinates after dimension conversion, and the results are summed within the neighborhood to obtain the displacement information of each pixel.

[0023] In one embodiment of the present invention, the optical flow estimator obtains optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining the preliminary intermediate frame image using the following formula:

[0024] Iwarp=M×warp(I1,Ft →1 )+(1-M).×warp(I2,Ft →2 );

[0025] Where Iwarp is the preliminary intermediate frame image; M is the mask generated at the same time when generating the optical flow, which is used to handle occlusions during the interpolation process; .× represents dot product; I1 and I2 are the first and second frames of the two images respectively; t is the interpolation time; Ft →1is the optical flow estimation from the intermediate frame to the first frame image; Ft →2 is the optical flow estimation from the intermediate frame to the second frame image; Ft →1 、Ft →2 and M constitute the optical flow estimation information; warp(·) is the operation of warping the image using optical flow.

[0026] In one embodiment of the present invention, the correction network corrects the preliminary intermediate frame image according to the multi-dimensional features and outputs a predicted intermediate frame image, including:

[0027] The correction network obtains the residual of the preliminary intermediate frame image according to the multi-dimensional features;

[0028] The preliminary intermediate frame image and the residual are summed to obtain a predicted intermediate frame image.

[0029] In one embodiment of the present invention, the first video dataset is a Vimeo90K dataset, and the second video dataset is a LAVIB dataset.

[0030] In one embodiment of the present invention, before pre-training the video interpolation model using a training strategy with random interpolation time and a preset multiple loss function, the method further includes:

[0031] Performing data enhancement processing on the first video dataset and setting training parameters;

[0032] Before optimizing and training the pre-trained video frame interpolation model, the method further includes:

[0033] Perform the data enhancement processing on the second video data set and set training parameters.

[0034] In one embodiment of the present invention, the multiple loss function is a weighted sum of a Laplace loss function and an optical flow loss function.

[0035] In a second aspect, an embodiment of the present invention provides a video frame insertion method, the video frame insertion method comprising:

[0036] Obtain the first frame image and the second frame image to be inserted;

[0037] The first frame image, the second frame image and the interpolation time are input into a pre-trained video interpolation model, and an inserted intermediate frame image is output; wherein the pre-trained video interpolation model is obtained according to the training method of the high-resolution video enhancement model based on continuous motion described in the first aspect.

[0038] The training method of the continuous motion-based high-resolution video enhancement model provided by the present invention has the following beneficial effects:

[0039] First, compared with the existing video interpolation model, the model proposed in this invention performs continuous motion modeling through correlation, ensuring the accuracy of motion information of intermediate frames, especially in large motion and complex scenes.

[0040] Second, the pre-training strategy of this invention enables the model to model continuous motion, making it more effective at handling video interpolation scenarios with large motion amplitudes. Further fine-tuning further improves the model’s performance in processing high-resolution, large-span videos.

[0041] Third, the model can not only handle video interpolation tasks with long time spans, but also maintain smooth and natural intermediate frame effects in high-resolution videos with often large motion amplitudes, making it suitable for various practical application scenarios.

[0042] By modeling continuous motion between frames, the present invention can effectively eliminate the uncertainty of motion between different frames, thereby enabling the model to accurately estimate the motion of specific positions between frames. Furthermore, based on accurate motion estimation, the present invention can achieve precise optical flow estimation and intermediate frame generation. This method improves the accuracy and effect of high-resolution video interpolation, especially when dealing with large-scale motion and complex scenes, with greater robustness and accuracy. The trained video interpolation model can be used in fields such as entertainment, digital media, and social public security, such as animation production, video acquisition, etc., to achieve accurate interpolation effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a method for training a high-resolution video enhancement model based on continuous motion provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the structure and principle of a video frame insertion model provided by an embodiment of the present invention;

[0045] Figure 3 A schematic flow chart of a video frame insertion method provided by an embodiment of the present invention;

[0046] Figure 4 This is the result of an embodiment of the present invention on a large-span video interpolation dataset;

[0047] Figure 5 The results of the embodiment of the present invention on high-resolution video and large-motion video;

[0048] Figure 6 These are the results of more large-span video interpolation test sets according to the embodiments of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0050] In order to solve the inter-frame motion uncertainty problem faced by existing high-resolution video interpolation methods, embodiments of the present invention provide a training method for a high-resolution video enhancement model based on continuous motion and a video interpolation method.

[0051] In a first aspect, an embodiment of the present invention provides a method for training a high-resolution video enhancement model based on continuous motion, such as Figure 1 As shown, the method may include the following steps S1 to S3:

[0052] S1, build a video frame insertion model;

[0053] The video interpolation model includes a feature extractor, a motion prior extractor, an optical flow estimator, and a correction network; the feature extractor is used to extract multi-dimensional features from two input frames; the motion prior extractor is used to obtain displacement information of the intermediate frame based on the high-dimensional features, correlation measurement, and interpolation time; the optical flow estimator is used to obtain optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining a preliminary intermediate frame image; the correction network is used to correct the preliminary intermediate frame image based on the multi-dimensional features and output a predicted intermediate frame image;

[0054] The structure of the video interpolation model can be found in Figure 2 The video interpolation model of the embodiment of the present invention is an improved design based on the overall model framework of the document "VFIMamba: Video Frame Interpolation with State Space Models". The model in the document includes a feature extractor, an optical flow estimator, and a correction network. The present invention adds a motion prior extractor to extract motion information to ensure the accuracy of the estimated optical flow.

[0055] The following describes each part of the video frame insertion model separately.

[0056] ① Feature Extractor

[0057] The feature extractor is used to extract multi-dimensional features from the two input frames. For ease of understanding, the two input frames are represented by I1 and I2. Figure 2The feature extractor includes three CNN modules and two Transformer modules, so the feature extractor can extract five-dimensional features from the input I1 and I2, wherein the two Transformer modules are used to output high-dimensional features.

[0058] ②Motion prior extractor

[0059] The motion prior extractor is used to obtain displacement information of the intermediate frame based on the high-dimensional features, correlation measurement and interpolation time. The process may include the following steps:

[0060] Step A1, using two motion prior extraction modules of the motion prior extractor to receive high-dimensional features of two frames of images;

[0061] from Figure 2 It can be seen that the motion prior extractor includes two motion prior extraction modules;

[0062] The two frames of images are respectively the first frame of image and the second frame of image, which are denoted by I1 and I2. It can be understood that I1 and I2 have the same size and the pixel positions are also one-to-one corresponding.

[0063] Step A2: for each pixel position, calculating a correlation value between the pixel value of the pixel position in the first frame image and the pixel values of each neighboring region of the pixel position in the second frame image based on the high-dimensional features of the two frames of image;

[0064] In video processing, motion information between frames is crucial for generating continuous and natural video interpolations. This paper extracts continuous motion priors to ensure that the model can effectively capture the motion pattern in the video, thereby generating smooth intermediate frames. First, the displacement information of each pixel in the image is estimated by calculating the correlation between frames. Correlation is a common method to measure the similarity between two frames of images. It can help identify which pixels have moved between the two frames and then estimate the displacement. The calculation method is shown in Formula 1:

[0065] Correlation(I1,I2) xymn =I1(x,y)I2(m,n) (1);

[0066] Here, I(x,y) represents the pixel value at the corresponding position. For example, I1(x,y) represents the pixel value at position (x,y) in I1. (m,n) represents the position of the pixel in the neighborhood of (x,y). Usually, the neighborhood is an area of size (k×k), where k can be 3, etc. Therefore, for I1(x,y), we can use the high-dimensional features of I1 and I2 and calculate a correlation value for each pixel in the (m,n) neighborhood of (x,y) in I2.

[0067] Suppose the length and width of images I1 and I2 are h and w respectively, that is, the images contain h×w pixels, that is, there are h×w positions (x, y). Through the above correlation calculation, each pixel position in I1 can obtain a correlation value with the k×k neighboring positions in I2. Then the size of Correlation(I1,I2) is (h,w,k,k).

[0068] Step A3, performing Softmax processing on the correlation value obtained at each pixel position;

[0069] To make displacement estimation more accurate, the present invention performs Softmax processing on the correlation value obtained at each pixel position to enhance the matching degree at the pixel level. During calculation, it is first transformed into a tensor of size (h,w,k×k), as shown in Formula 2:

[0070]

[0071] Where i represents the serial number of the pixel position;

[0072] Softmax can give more relevant pixel displacements greater weights.

[0073] Step A4, obtaining the displacement information of each pixel based on the correlation value after Softmax processing and the coordinates of each pixel position;

[0074] Specifically, step A4 may include:

[0075] Step A41, performing tensor transformation on the correlation value after Softmax processing;

[0076] This step transforms Correlation(I1,I2) back into a tensor of size (h,w,k,k).

[0077] In step A42, for each pixel position, after mapping its coordinates from two dimensions to the same dimension as the feature, the current correlation value of the pixel position is multiplied by its coordinates after dimension conversion, and the results are summed within the neighborhood to obtain the displacement information of each pixel.

[0078] Since the motion prior extractor extracts motion information at the feature level, the coordinates used to calculate the displacement information are first mapped from two dimensions to the same dimension as the feature through a linear mapping layer, and then the displacement information is calculated using Formula 3.

[0079]

[0080] In Formula 3, (x, y) represents the pixel position coordinates after mapping from two dimensions to the same dimension as the feature. Δx(x, y) represents the displacement information of the corresponding pixel.

[0081] Step A5: Obtain the displacement information of the corresponding pixel in the intermediate frame according to the product of the displacement information of each pixel and the interpolation time.

[0082] Obtaining the displacement information of each pixel means obtaining the displacement information between frames, and then adjusting the displacement information according to the interpolation time.

[0083] Assuming that the interpolation time is t (a value between 0 and 1), the displacement information of the corresponding pixel finally interpolated is the product of the displacement information of the pixel obtained in step A4 and the interpolation time, as shown in Formula 4:

[0084] Δxmid = Δx·t (4);

[0085] Wherein, Δx is the displacement information of the pixel (x, y) obtained in step A4, and Δxmid represents the displacement information of the corresponding pixel in the intermediate frame.

[0086] Through this process, the embodiment of the present invention can generate and model different priors for different interpolation frame times t.

[0087] ③Optical flow estimator

[0088] Based on the above prior, we can use this prior to calculate the intermediate frame optical flow Ft →1 、Ft →2 Compared with the previous method of estimating the optical flow of intermediate frames using features, the estimation of the optical flow of intermediate frames using this prior has stronger interpretability and stronger robustness when performing multi-frame insertion. The intermediate frames can be preliminarily obtained through the optical flow of intermediate frames.

[0089] The optical flow estimator obtains optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining the preliminary intermediate frame image using the formula:

[0090] Iwarp=M×warp(I1,Ft →1 )+(1-M).×warp(I2,Ft →2 ) (5);

[0091] Where Iwarp is the preliminary intermediate frame image; M is the mask generated at the same time when generating the optical flow, which is used to handle occlusions during the interpolation process; .× represents dot product; I1 and I2 are the first and second frames of the two images respectively; t is the interpolation time; Ft →1 is the optical flow estimation from the intermediate frame to the first frame image; Ft →2 is the optical flow estimation from the intermediate frame to the second frame image; Ft →1 、Ft →2 and M constitute the optical flow estimation information, Figure 2 F is used in the above equations; warp(·) is an operation to distort the image using optical flow.

[0092] ④Correction network

[0093] The correction network corrects the preliminary intermediate frame image according to the multi-dimensional features and outputs a predicted intermediate frame image, comprising:

[0094] 1) The correction network obtains the residual of the preliminary intermediate frame image according to the multi-dimensional features;

[0095] See Figure 2 The correction network includes multiple CNN modules, and the first five CNN modules each receive one of the multi-dimensional features. Through the processing of the multiple CNN modules of the correction network, the residual ΔI of the preliminary intermediate frame image is output.

[0096] 2) The preliminary intermediate frame image and the residual are summed to obtain a predicted intermediate frame image.

[0097] Then, the preliminary intermediate frame image Iwarp and the residual ΔI are summed. For the summation process, see Figure 2 As shown in the adder in , the predicted intermediate frame image Ipred is obtained.

[0098] For the specific structure of the feature extractor, optical flow estimator, and correction network, please refer to the relevant literature.

[0099] S2, pre-training the video interpolation model based on the first video dataset using a training strategy with random interpolation time and a preset multiple loss function; wherein the multiple loss function is constructed based on a Laplace loss function and an optical flow loss function;

[0100] In S1, a video interpolation model was constructed. In S2, the present invention proposed a universal pre-training strategy, enabling the model to have stronger continuous motion modeling capabilities during the training phase. To effectively handle the uncertainty of inter-frame displacement in large-scale motion and high-resolution videos, the Vimeo90K dataset was selected as the first video dataset, and multiple loss functions were designed to optimize the model's motion modeling.

[0101] Specifically, we first introduce Laplace Loss to optimize the pixel difference between the generated intermediate frame and the real frame. The formula is:

[0102]

[0103] Where Ipred is the intermediate frame generated by the model, and Igt is the real intermediate frame. L represents the average pooling operation, i represents the number of average pooling operations, and N represents the maximum number of average pooling operations. The value of N can be set as needed, for example, 5.

[0104] The role of the Laplace loss function is to maintain the structural consistency between the real frame and the predicted frame.

[0105] In addition, the present invention also designs a loss function based on optical flow to further improve the quality of the generated frames by optimizing the consistency of motion between frames. For the displacement of each pixel, the model needs to infer the direction of motion through optical flow and minimize the difference between the predicted displacement and the actual displacement. The optical flow loss function is as follows:

[0106]

[0107] Optical flow loss further enhances the model's motion modeling capabilities, especially in large motion or complex scenes, and can maintain motion consistency, resulting in more natural interpolation effects.

[0108] The multiple loss function is a weighted sum of the Laplace loss function and the optical flow loss function, and the weights are set as needed.

[0109] Optionally, before pre-training the video interpolation model using a training strategy with random interpolation time and a preset multiple loss function, the training method for the continuous motion-based high-resolution video enhancement model further includes:

[0110] Performing data enhancement processing on the first video dataset and setting training parameters;

[0111] Specifically, during the training process, the Vimeo90K dataset was used for data augmentation. During each training session, three frames of video were randomly selected, and 256×256 image blocks were cropped from the three frames. Data augmentation was then performed, including random flipping, time reversal, and random rotation. The two frames before and after were used as the two frames to be interpolated, and the middle frame was used as the actual middle frame, i.e., the label data. The training parameters were set as follows: the training batch size was set to 32, the optimizer used was AdamW, the training cycle was set to 300, and the learning rate was increased from 1e-1 using the cosine annealing strategy. 4 Gradually decays to 1e- 5.

[0112] Compared to the previous approach of directly interpolating three consecutive frames at time t = 0.5, the pre-training strategy of this invention uses random interpolation times. This enables the model to model continuous motion and enhances its performance for high-resolution videos and videos with large motion. The improvement achieved by the pre-training method proposed in this invention can be seen in Table 4 of the experiments below.

[0113] The specific training process can be understood by referring to the conventional neural network training process, which will not be explained in detail here.

[0114] S3, based on a second video dataset containing long high-resolution video clips, optimizing and training the pre-trained video interpolation model to obtain a trained video interpolation model; the trained video interpolation model is used to output an interpolated intermediate frame image for two input frames.

[0115] To further improve the model's interpolation capabilities over long time spans, we fine-tune the model using a long video and high-resolution dataset, the LAVIB (Large-scale Video Interpolation Benchmark) dataset (referenced in "Alexandros Stergiou. Lavib: Large-scale video interpolation benchmark. In NeurIPS, 2024"). The LAVIB dataset contains video clips with longer time spans. We extract a continuous 20-frame segment from each video clip and randomly select three frames from it for training.

[0116] Similarly, before optimizing the pre-trained video frame interpolation model, the method further includes:

[0117] Perform the data enhancement processing on the second video data set and set training parameters.

[0118] Specifically, each frame is processed by data augmentation operations such as random scaling and cropping. Each frame is scaled to 256×256, the training batch size is set to 16, and the learning rate is set from 1e- 5 Starts and decays to 1e- 6 The training settings of the fine-tuning stage are the same as those of pre-training. By fine-tuning on longer video clips, the model can capture motion information over a longer time span.

[0119] The loss function definition remains the same as in the pre-training phase, with the goal still being to optimize the model's inter-frame interpolation capabilities by minimizing the weighted sum of the Laplacian loss and the optical flow loss. Fine-tuning enables the model to handle more complex video interpolation tasks, particularly in high-resolution video interpolation, generating high-quality intermediate frames that are smoother and more natural.

[0120] Compared to previous methods, fine-tuning the model specifically on long, high-resolution videos further improves its performance in modeling continuous large-scale motion. This can be seen in the results in Table 1 of the experiments below. Furthermore, fine-tuning significantly reduces the computational effort required for model training compared to direct training on videos with large motion. This approach achieves excellent results on large-scale motion datasets.

[0121] The training method of the continuous motion-based high-resolution video enhancement model provided by the embodiment of the present invention has the following beneficial effects:

[0122] First, compared with the existing video interpolation model, the model proposed in this invention performs continuous motion modeling through correlation, ensuring the accuracy of motion information of intermediate frames, especially in large motion and complex scenes.

[0123] Second, the pre-training strategy of this invention enables the model to model continuous motion, making it more effective at handling video interpolation scenarios with large motion amplitudes. Further fine-tuning further improves the model’s performance in processing high-resolution, large-span videos.

[0124] Third, the model can not only handle video interpolation tasks with long time spans, but also maintain smooth and natural intermediate frame effects in high-resolution videos with often large motion amplitudes, making it suitable for various practical application scenarios.

[0125] By modeling continuous motion between frames, the present invention can effectively eliminate the uncertainty of motion between different frames, thereby enabling the model to accurately estimate the motion of specific positions between frames. Furthermore, based on accurate motion estimation, the present invention can achieve precise optical flow estimation and intermediate frame generation. This method improves the accuracy and effectiveness of high-resolution video interpolation, especially when dealing with large-scale motion and complex scenes, with greater robustness and accuracy. The trained video interpolation model can be used in fields such as entertainment, digital media, and social public security, such as animation production and video acquisition.

[0126] In the second aspect, corresponding to the embodiment of the training method of the high-resolution video enhancement model based on continuous motion described in the first aspect, the embodiment of the present invention further provides a video frame insertion method, such as Figure 3 As shown, the video frame insertion method includes:

[0127] S100, obtaining a first frame image and a second frame image to be inserted;

[0128] S200: Input the first frame image, the second frame image, and the interpolation time into a pre-trained video interpolation model, and output an inserted intermediate frame image; wherein the pre-trained video interpolation model is obtained according to the training method of the high-resolution video enhancement model based on continuous motion described in the first aspect.

[0129] The present invention can output high-resolution intermediate frames from low-resolution videos input by a pre-trained and fine-tuned video interpolation model. These intermediate frames are not only visually similar to the real frames, but also maintain a high degree of consistency in motion information.

[0130] For the structure and processing of the pre-trained video interpolation model, please refer to the relevant content in the first aspect and will not be repeated here.

[0131] The video interpolation method provided in the embodiment of the present invention is completed using a pre-trained video interpolation model. It can accurately estimate the motion of specific positions between frames, achieve precise optical flow estimation and intermediate frame generation, and improve the accuracy and effect of high-resolution video interpolation. In particular, it has greater robustness and accuracy when processing large-scale motion and complex scenes.

[0132] 1. Simulation conditions

[0133] The experiment of the present invention was carried out on an Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHz, an NVIDIA RTX 2080Ti, and an Ubuntu 16.04 operating system. The programming language used was Python, the deep learning network framework used was Pytorch, and the training data sets used were Vimeo90K and LAVIB.

[0134] To fairly and objectively evaluate the performance of our methods, we ensured that all comparison models were trained using the same training dataset. We also trained other state-of-the-art algorithms using the same settings for comparison. Specifically including UPRNET, VFIMamba, SGM-VFI, and AMT, their references are "Xin Jin, Longhai Wu, JieChen, Youxin Chen, Jayoon Koo, and Cheul-Hee Hahm. Aunified pyramid recurrent network for video frame interpolation. In 2023 IEEE / CVF Conference on ComputerVision and Pattern Recognition (CVPR), pages 1578-1587,2023","Guozhen Zhang,Chunxu Liu,Yutao Cui,Xiaotong Zhao,Kai Ma,and Limin Wang.VFIMamba: Video FrameInterpolation with State Space Models,2024.","Chunxu Liu,Guozhen Zhang,RuiZhao,and Limin Wang.Sparse global matching for video frame interpolation with large motion.In 2024IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR),pages 19125-19134,Los Alamitos, CA, USA, 2024" and "Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, ChunLe Guo, and Ming-Ming Cheng. Amt: All-pairsmulti-field transforms for efficient frame interpolation. In 2023 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), pages9801-9810, 2023."In addition to the retrained algorithms, the experiments in this paper also compared the models provided by themselves and the models provided by some other algorithms including DQBC (IJCAI 2023) and RIFE (ECCV 2022).

[0135] 2. Simulation content

[0136] Table 1 Performance of each model on large-span high-resolution videos

[0137]

[0138]

[0139] In Table 1, “-RT” indicates the model trained according to the training settings proposed in the present invention, “V” indicates Vimeo90K, and “L” indicates LAVIB;

[0140] Table 2 Performance of each model on large motion videos

[0141]

[0142] In Table 2, “-RT” indicates the retrained model of the present invention, “-PT” indicates the pretrained model, “V” indicates Vimeo90K, and “L” indicates LAVIB.

[0143] Table 3 Performance of each model on high-resolution videos

[0144]

[0145]

[0146] In Table 3, “-RT” indicates the retrained model of the present invention, “-PT” indicates the model trained under the pre-training strategy, “V” indicates Vimeo90K, and “L” indicates LAVIB.

[0147] It can be seen from Tables 1, 2, and 3 that the method of the present invention is much better than the existing methods in objective evaluation indicators PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), and the retrained method of the present invention shows greater superiority in large motion video interpolation and high-resolution video interpolation tasks compared with the weights provided by its authors. From the visual effect point of view, other existing methods may have serious distortion in a certain local position, while the present invention can maintain its general shape. At the same time, in order to verify the superiority of the pre-training method proposed by the present invention, the results of the pre-training algorithm and the comparison with the best existing method are also given in Table 4. It can be seen that after pre-training by the present invention, the existing methods can surpass the best existing methods to a certain extent. In addition, please see the visual evaluation results of each model on large-span high-resolution videos. Figures 4 to 6 shown. Figure 4 To show the results on the large span video interpolation dataset, Figure 5 For high-resolution and large-motion video results, Figure 6 Results on the test set for more large-span video interpolation frames.

[0148] Table 4 Comparison of each model after pre-training with the best existing method VFIMamba

[0149]

[0150] In Table 4, “-FT” represents the fine-tuned model of the present invention, “-PT” represents the model trained according to the pre-training strategy of the present invention, “V” represents Vimeo90K, and “L” represents LAVIB.

[0151] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A training method for a high-resolution video enhancement model based on continuous motion, characterized in that: include: Constructing a video interpolation model; wherein the video interpolation model includes a feature extractor, a motion prior extractor, an optical flow estimator, and a correction network; the feature extractor is used to extract multi-dimensional features from two input frames; the motion prior extractor is used to obtain displacement information of the intermediate frame based on the extracted high-dimensional features, a correlation measure, and the interpolation time; the optical flow estimator is used to obtain optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining a preliminary intermediate frame image; the correction network is used to correct the preliminary intermediate frame image based on the multi-dimensional features and output a predicted intermediate frame image; Based on the first video dataset, the video interpolation model is pre-trained using a training strategy with random interpolation time and a preset multiple loss function; wherein the multiple loss function is constructed based on a Laplace loss function and an optical flow loss function; Based on a second video dataset containing long high-resolution video clips, the pre-trained video interpolation model is optimized and trained to obtain a trained video interpolation model; the trained video interpolation model is used to output an interpolated intermediate frame image for two input frames.

2. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The feature extractor includes three CNN modules and two Transformer modules; wherein the two Transformer modules are used to output high-dimensional features.

3. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The process of obtaining the displacement information of the intermediate frame by the motion prior extractor based on the high-dimensional features output by the feature extractor and the correlation measurement and the interpolation time includes: Using the two motion prior extraction modules of the motion prior extractor, high-dimensional features of two frames of images are received; wherein the two frames of images are a first frame of image and a second frame of image; For each pixel position, calculating a correlation value between the pixel value of the pixel position in the first frame of image and the pixel values of each neighboring region of the pixel position in the second frame of image based on the high-dimensional features of the two frames of image; The correlation value obtained at each pixel position is processed by Softmax; According to the correlation value after Softmax processing and the coordinates of each pixel position, the displacement information of each pixel is obtained; The displacement information of the corresponding pixel in the intermediate frame is obtained according to the product of the displacement information of each pixel and the interpolation time.

4. The method for training a high-resolution video enhancement model based on continuous motion according to claim 3, wherein: The displacement information of each pixel is obtained based on the correlation value after Softmax processing and the coordinates of each pixel position, including: Perform tensor transformation on the correlation value after Softmax processing; For each pixel position, after mapping its coordinates from two dimensions to the same dimension as the feature, the current correlation value of the pixel position is multiplied by its coordinates after dimension conversion, and the results are summed within the neighborhood to obtain the displacement information of each pixel.

5. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The optical flow estimator obtains optical flow estimation information based on the displacement information of the intermediate frame, thereby obtaining the preliminary intermediate frame image using the formula: Iwarp=M.×warp(I1,Ft →1 )+(1-M).×warp(I2,Ft →2 ); Where Iwarp is the preliminary intermediate frame image; M is the mask generated at the same time when generating the optical flow, which is used to handle occlusions during the interpolation process; .× represents dot product; I1 and I2 are the first and second frames of the two images respectively; t is the interpolation time; Ft →1 is the optical flow estimation from the intermediate frame to the first frame image; Ft →2 is the optical flow estimation from the intermediate frame to the second frame image; Ft →1 、Ft →2 and M constitute the optical flow estimation information; warp(·) is the operation of warping the image using optical flow.

6. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The correction network corrects the preliminary intermediate frame image according to the multi-dimensional features and outputs a predicted intermediate frame image, comprising: The correction network obtains the residual of the preliminary intermediate frame image according to the multi-dimensional features; The preliminary intermediate frame image and the residual are summed to obtain a predicted intermediate frame image.

7. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The first video dataset is a Vimeo90K dataset, and the second video dataset is a LAVIB dataset.

8. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: Before pre-training the video frame insertion model using a training strategy with random frame insertion time and a preset multiple loss function, the method further includes: Performing data enhancement processing on the first video dataset and setting training parameters; Before optimizing and training the pre-trained video frame interpolation model, the method further includes: Perform the data enhancement processing on the second video data set and set training parameters.

9. The method for training a high-resolution video enhancement model based on continuous motion according to claim 1, wherein: The multiple loss function is a weighted sum of the Laplace loss function and the optical flow loss function.

10. A video frame insertion method, characterized in that: include: Obtain the first frame image and the second frame image to be inserted; The first frame image, the second frame image and the interpolation time are input into a pre-trained video interpolation model, and an inserted intermediate frame image is output; wherein the pre-trained video interpolation model is obtained according to the training method of the continuous motion-based high-resolution video enhancement model according to any one of claims 1 to 9.

Images