Video frame interpolation methods, apparatus, computer equipment, and storage media

Abstract

This application relates to a video frame interpolation method, apparatus, computer device, storage medium, and computer program product. The method includes: inputting a first video frame and a second video frame into a pre-trained frame interpolation model to obtain an intermediate frame between the first and second video frames; inputting a downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame into a pre-trained scale combination prediction model to obtain predicted scale combination information of the pre-trained frame interpolation model; adjusting the current scale combination information of the pre-trained frame interpolation model according to the predicted scale combination information to obtain an adjusted frame interpolation model; using the adjusted frame interpolation model as the corresponding pre-trained frame interpolation model, repeating the above steps until a preset termination condition is met, and using the intermediate frame corresponding to the satisfied preset termination condition as the target video frame between the first and second video frames. This method can improve the quality of the generated intermediate frames.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video frame interpolation method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of computer vision technology, people have increasingly higher requirements for video frame rates. Therefore, frame interpolation technology is becoming more and more important in video processing. For example, video frame interpolation technology can be used for video compression, slow-motion processing, and replacing bad frames in the original video.

[0003] In traditional techniques, the intermediate frame between two video frames is usually estimated based on the optical flow of two adjacent video frames. However, when optical flow synthesis is performed on intermediate frames whose video content involves large-scale motion, the resulting intermediate frame is prone to loss of video details, making it blurry and of low quality. Summary of the Invention

[0004] Therefore, it is necessary to provide a video frame interpolation method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the quality of generated intermediate frames in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a video frame interpolation method. The method includes:

[0006] The first video frame and the second video frame are input into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0007] The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model.

[0008] Based on the predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model.

[0009] The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the satisfying of the preset termination condition is used as the target video frame between the first video frame and the second video frame.

[0010] Secondly, this application also provides a video frame interpolation apparatus. The apparatus includes:

[0011] The intermediate frame acquisition module is used to input the first video frame and the second video frame into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0012] The scale combination prediction module is used to input the downsampled video frame obtained based on the first video frame, the intermediate frame and the second video frame into the pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained frame interpolation model.

[0013] The frame interpolation model adjustment module is used to adjust the current scale combination information of the pre-trained frame interpolation model according to the prediction scale combination information to obtain the adjusted frame interpolation model.

[0014] The video frame determination module is used to take the adjusted frame interpolation model as the pre-trained frame interpolation model, and jump to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is taken as the target video frame between the first video frame and the second video frame.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] The first video frame and the second video frame are input into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0017] The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model.

[0018] Based on the predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model.

[0019] The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the satisfying of the preset termination condition is used as the target video frame between the first video frame and the second video frame.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] The first video frame and the second video frame are input into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0022] The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model.

[0023] Based on the predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model.

[0024] The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the satisfying of the preset termination condition is used as the target video frame between the first video frame and the second video frame.

[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0026] The first video frame and the second video frame are input into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0027] The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model.

[0028] Based on the predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model.

[0029] The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the satisfying of the preset termination condition is used as the target video frame between the first video frame and the second video frame.

[0030] The aforementioned video frame interpolation method, apparatus, computer equipment, storage medium, and computer program product input a first video frame and a second video frame into a pre-trained frame interpolation model to obtain an intermediate frame between the first and second video frames; input a downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame into a pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained frame interpolation model; adjust the current scale combination information of the pre-trained frame interpolation model according to the predicted scale combination information to obtain an adjusted frame interpolation model; use the adjusted frame interpolation model as the corresponding pre-trained frame interpolation model, and jump to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain an intermediate frame between the first and second video frames, until a preset termination condition is met, and use the intermediate frame corresponding to the satisfied preset termination condition as the target video frame between the first and second video frames. This method inputs downsampled video frames into a pre-trained scale combination prediction model. This not only enables the acquisition of predicted scale combination information using video frames smaller than the first, intermediate, and second video frames, but also reduces the computational load of the pre-trained scale combination prediction model, improving processing efficiency. By adjusting the current scale combination information in the pre-trained frame interpolation model using the pre-trained scale combination prediction model, the intermediate frames can be well fitted regardless of whether the video frame has a large or small motion amplitude, thereby improving the quality of the generated target intermediate frames. Attached Figure Description

[0031] Figure 1 This is a flowchart of a video frame interpolation method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating the steps of training a pre-trained frame interpolation model in one embodiment.

[0033] Figure 3 This is a flowchart illustrating the steps of obtaining the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample in one embodiment.

[0034] Figure 4 This is a flowchart illustrating a video frame interpolation method in another embodiment;

[0035] Figure 5 This is a flowchart illustrating the video frame interpolation method in yet another embodiment;

[0036] Figure 6 This is a schematic diagram of the structure of the frame interpolation model to be trained in one embodiment;

[0037] Figure 7This is a schematic diagram of the structure of a pre-trained scale combination prediction model in one embodiment;

[0038] Figure 8 This is a schematic diagram illustrating the principle of the video frame interpolation method in yet another embodiment;

[0039] Figure 9 This is a structural block diagram of a video frame interpolation device in one embodiment;

[0040] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] In one embodiment, such as Figure 1 As shown, a video frame interpolation method is provided. This embodiment illustrates the method applied to a server, but it is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The video frames, such as the first video frame and the second video frame, can be pre-stored in the server or sent to the server by the terminal device. In this embodiment, the method includes the following steps:

[0043] Step S101: Input the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0044] The pre-trained frame interpolation model refers to the model used to generate intermediate frames between the first video frame and the second video frame. This model can be, but is not limited to, a multi-scale deep learning frame interpolation model such as a convolutional neural network.

[0045] The first video frame, the intermediate frame, and the second video frame are three consecutive video frames in the video; the first video frame refers to the video frame preceding the intermediate frame; and the second video frame refers to the video frame following the intermediate frame.

[0046] Specifically, when the server acquires the first and second video frames from the target video, it can input them into a pre-trained frame interpolation model. The pre-trained frame interpolation model outputs the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame. Based on the first video frame, the second video frame, the first optical flow, and the second optical flow, an intermediate frame between the first and second video frames is synthesized. Here, the first optical flow refers to the optical flow from the intermediate frame to the first video frame, and the second optical flow refers to the optical flow from the intermediate frame to the second video frame.

[0047] Step S102: Input the downsampled video frame obtained based on the first video frame, intermediate frame and second video frame into the pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model.

[0048] The pre-trained scale combination prediction model refers to a model used to predict the scaling scale of video frames, which can be, but is not limited to, a reinforcement learning model.

[0049] Specifically, the server concatenates the first video frame, intermediate frames, and second video frames to obtain a concatenated video frame. This concatenated video frame is then downsampled to obtain a downsampled video frame. The downsampled video frame is then used as a state variable input into a pre-trained scale combination prediction model, which predicts the scale combination information. This predicted scale combination information is obtained by combining multiple scaling scales.

[0050] Step S103: Adjust the current scale combination information of the pre-trained frame interpolation model according to the predicted scale combination information to obtain the adjusted frame interpolation model.

[0051] In the case where the pre-trained interpolation model generates the intermediate frame between the first video frame and the second video frame for the first time, that is, before the pre-trained scale combination prediction model has predicted the predicted scale combination information of the pre-trained interpolation model, the current scale combination information of the pre-trained interpolation model can be preset or can be the empirical value obtained by the pre-trained interpolation model during the training process.

[0052] Specifically, after obtaining the predicted scale combination information, the server can input it into a pre-trained frame interpolation model so that the current scale combination information in the pre-trained frame interpolation model can be updated subsequently, thus obtaining an adjusted frame interpolation model. The predicted scale combination information includes the number of scaling scales.

[0053] Step S104: The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until the preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is used as the target video frame between the first video frame and the second video frame.

[0054] The target video frame refers to the video frame between the first and second video frames that generate the best quality within the preset termination conditions.

[0055] Specifically, after obtaining the adjusted frame interpolation model, the server can execute step S101 again, that is, input the first video frame and the second video frame into the adjusted frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame; then execute step S102 again to obtain the prediction scale combination information of the adjusted frame interpolation model; then execute step S103 again to update the adjusted frame interpolation model; repeat steps S101 to S103 until a preset termination condition is met. The intermediate frame output by the pre-trained frame interpolation model when the preset termination condition is met is taken as the target video frame between the first video frame and the second video frame.

[0056] In the above video interpolation method, the first video frame and the second video frame are input into a pre-trained interpolation model to obtain an intermediate frame between the first video frame and the second video frame; the downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained interpolation model; the current scale combination information of the pre-trained interpolation model is adjusted according to the predicted scale combination information to obtain an adjusted interpolation model; the adjusted interpolation model is used as the corresponding pre-trained interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained interpolation model to obtain an intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is used as the target video frame between the first video frame and the second video frame. This method inputs downsampled video frames into a pre-trained scale combination prediction model. This not only enables the acquisition of predicted scale combination information using video frames smaller than the first, intermediate, and second video frames, but also reduces the computational load of the pre-trained scale combination prediction model, improving processing efficiency. By adjusting the current scale combination information in the pre-trained frame interpolation model using the pre-trained scale combination prediction model, the intermediate frames can be well fitted regardless of whether the video frame has a large or small motion amplitude, thereby improving the quality of the generated target intermediate frames.

[0057] In one embodiment, such as Figure 2 As shown, the pre-trained frame interpolation model can be trained in the following way:

[0058] Step S201: Input the first video frame sample and the second video frame sample into the frame interpolation model to be trained to obtain the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample.

[0059] The frame interpolation model to be trained includes a first optical flow residual prediction model, a second optical flow residual prediction model, and an optical flow residual correction network. The first optical flow residual prediction model, the second optical flow residual prediction model, and the optical flow residual correction network can be obtained by combining multiple convolutional neural networks.

[0060] The second optical flow residual prediction model contains at least two optical flow residual prediction models. These two models have different model parameters, but the model structures are the same. For example, the two optical flow residual prediction models can be convolutional neural networks with the same network structure, but they use different hyperparameters.

[0061] Specifically, the server extracts three consecutive video frames from the sample video, labeling them as the first initial video frame sample, the actual initial intermediate frame sample, and the second initial video frame sample. The first initial video frame sample, the actual initial intermediate frame sample, and the second initial video frame sample are then normalized to obtain normalized first initial video frame samples, normalized actual initial intermediate frame samples, and normalized second initial video frame samples. Color inversion, image inversion, and segmentation processing can then be applied to the normalized first initial video frame samples, normalized actual initial intermediate frame samples, and normalized second initial video frame samples to obtain the first video frame sample, the actual initial intermediate frame sample, and the second video frame sample.

[0062] For example, three consecutive video frames are extracted from a sample video and labeled A, B, and C. These three frames (A, B, and C) are then normalized to a range of 0 to 1 by dividing by 255. The red, green, and blue channels of the normalized A, B, and C can be randomly inverted. The video frames can also be randomly inverted vertically, horizontally, or vertically, and a portion can be randomly cut. The resulting first video frame sample, the actual initial intermediate frame sample, and the second video frame sample are used to train the interpolation model.

[0063] Furthermore, the server processes the first video frame sample and the second video frame sample using the first optical flow residual prediction model in the frame interpolation model to be trained, to obtain the optical flow residual corresponding to the first video frame sample and the optical flow residual corresponding to the second video frame sample. Then, using two second optical flow residual prediction models with different model parameters but the same model structure, the server processes the first video frame sample, the second video frame sample, the optical flow residual corresponding to the first video frame sample, and the optical flow residual corresponding to the second video frame sample twice in sequence, finally obtaining the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample, so as to obtain a more accurate target optical flow.

[0064] Step S202: Based on the first video frame sample, the second video frame sample, the actual intermediate frame sample between the first video frame sample and the second video frame sample, the first target optical flow and the second target optical flow, obtain the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample.

[0065] Here, the actual intermediate frame sample refers to the real intermediate frame between the first video frame sample and the second video frame sample. For example, if there are three consecutive video frames in the target video, namely A, B and C, then A can be regarded as the first video frame sample, C can be regarded as the second video frame sample, and B can be regarded as the actual intermediate frame sample; B' obtained according to the above steps S101 to S104 can be regarded as the target intermediate frame between A and C.

[0066] The third target optical flow refers to the optical flow predicted by the optical flow residual correction network for the first video frame sample after learning the actual intermediate frame samples; the fourth target optical flow refers to the optical flow predicted by the optical flow residual correction network for the second video frame sample after learning the actual intermediate frame samples.

[0067] Specifically, the server processes a video frame sample, a second video frame sample, an actual intermediate frame sample between the first and second video frame samples, a first target optical flow, and a second target optical flow through an optical flow residual correction network to obtain a third target optical flow corresponding to the first video frame sample and a fourth target optical flow corresponding to the second video frame sample.

[0068] Step S203: Based on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow, obtain the loss function of the frame interpolation model to be trained.

[0069] Step S204: Based on the loss function, iteratively train the interpolation model to be trained to obtain the pre-trained interpolation model.

[0070] Specifically, the server performs regularization and Laplacian processing on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow to obtain the loss function of the interpolation model to be trained. Then, the first optical flow residual prediction model, the second optical flow residual prediction model, and the optical flow residual correction network in the interpolation model to be trained are updated with gradients using this loss function to obtain the pre-trained interpolation model.

[0071] In this embodiment, the first optical flow residual prediction model and two second optical flow residual prediction models with different parameters but the same model structure are used to process the first video frame sample, the second video frame sample, the first video frame sample, and the second video frame sample in sequence, which can improve the accuracy of the target optical flow. The optical flow residual correction network is trained using the first video frame sample, the second video frame sample, and the actual intermediate frame sample, so that the optical flow residual correction network can learn the optical flow corresponding to the first and second video frame samples when the actual intermediate frame samples are used. The loss function is calculated based on the optical flow residual learned by the optical flow residual correction network, and the frame interpolation model to be trained is iteratively trained using the loss function to obtain the pre-trained frame interpolation model, which greatly improves the prediction accuracy of the pre-trained frame interpolation model for optical flow residual, thereby improving the quality of the target intermediate frame.

[0072] In one embodiment, such as Figure 3 As shown, the first video frame sample and the second video frame sample are input into the frame interpolation model to be trained to obtain the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample. Specifically, this includes the following:

[0073] Step S301: Based on the first preset scale in the preset scale combination information of the frame interpolation model to be trained, the first video frame sample and the second video frame sample are scaled to obtain the first scaled sample and the second scaled sample.

[0074] Step S302: Input the first scaled sample and the second scaled sample into the first optical flow residual prediction model in the frame interpolation model to be trained, to obtain the first initial optical flow residual corresponding to the first scaled sample and the second initial optical flow residual corresponding to the second scaled sample, and use the first initial optical flow residual as the first initial optical flow and the second initial optical flow residual as the second initial optical flow.

[0075] The preset scale combination information includes the video frame scaling scale corresponding to each model in the frame interpolation model to be trained. For example, the first preset scale corresponding to the first optical flow residual prediction model, and the second and third preset scales corresponding to the second optical flow residual prediction model (which includes two models with different model parameters but the same model structure).

[0076] Specifically, the server scales the first video frame sample according to a first preset scale to obtain a first scaled sample; it also scales the second video frame sample according to the first preset scale to obtain a second scaled sample; the first and second scaled samples are input into the first optical flow residual prediction model in the frame interpolation model to be trained to obtain the first initial optical flow residual corresponding to the first scaled sample and the second initial optical flow residual corresponding to the second scaled sample; it should be noted that, except for computer animation and some special scenes, the true value of the optical flow of the video frame cannot be obtained, so the optical flow of the first and second video frame samples can both be set to 0; the first initial optical flow residual is added to the optical flow of the first video frame sample to obtain the first initial optical flow, that is, the first initial optical flow equals the first initial optical flow residual; the second initial optical flow residual is added to the optical flow of the second video frame sample to obtain the second initial optical flow, that is, the second initial optical flow equals the second initial optical flow residual.

[0077] Step S303: According to the second preset scale in the preset scale combination information, the first video frame sample and the second video frame sample are scaled to obtain the third scaled sample and the fourth scaled sample.

[0078] Step S304: Input the third scaled sample, the fourth scaled sample, the first initial optical flow, and the second initial optical flow into the second optical flow residual prediction model in the frame interpolation model to be trained, to obtain the first updated optical flow residual of the first initial optical flow and the second updated optical flow residual of the second initial optical flow; the first updated optical flow residual is used to update the first initial optical flow to obtain the first updated optical flow; the second updated optical flow residual is used to update the second initial optical flow to obtain the second updated optical flow.

[0079] Specifically, the server scales the first video frame sample according to a second preset scale to obtain a third scaled sample; it scales the second video frame sample according to the second preset scale to obtain a fourth scaled sample; the third scaled sample, the fourth scaled sample, the first initial optical flow, and the second initial optical flow are input into a second optical flow residual prediction model in the frame interpolation model to be trained to obtain a first updated optical flow residual and a second updated optical flow residual; the first initial optical flow and the first updated optical flow residual are added to obtain a first updated optical flow; the second initial optical flow and the second updated optical flow residual are added to obtain a second updated optical flow.

[0080] Step S305: According to the third preset scale in the preset scale combination information, the first video frame sample and the second video frame sample are scaled to obtain the fifth scaled sample and the sixth scaled sample.

[0081] Step S306: Input the fifth scaled sample, the sixth scaled sample, the first updated optical flow, and the second updated optical flow into the second optical flow residual prediction model to obtain the first target optical flow residual of the first updated optical flow and the second target optical flow residual of the second updated optical flow.

[0082] Step S307: Based on the first target optical flow residual and the second target optical flow residual, update the first update optical flow and the second update optical flow accordingly to obtain the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample.

[0083] It should be noted that the second optical flow residual prediction models in steps S304 and S306 are two models with different parameters but the same model structure. Furthermore, the first initial optical flow, the first updated optical flow, and the first target optical flow all refer to the optical flow from the intermediate frame to the first video frame; the second initial optical flow, the second updated optical flow, and the second target optical flow all refer to the optical flow from the intermediate frame to the second video frame.

[0084] Specifically, the server scales the first video frame sample according to a third preset scale to obtain a fifth scaled sample; it then scales the second video frame sample according to the same third preset scale to obtain a sixth scaled sample. The fifth and sixth scaled samples, along with the first and second updated optical flows, are input into another second optical flow residual prediction model within the frame interpolation model to be trained, to obtain the first and second target optical flow residuals. The first target optical flow residual is added to the first updated optical flow to obtain the first target optical flow; and the second target optical flow residual is added to the second updated optical flow to obtain the second target optical flow. The third preset scale can be set to 0, meaning that no scaling is applied to the first and second video frame samples.

[0085] In this embodiment, the first video frame sample and the second video frame sample are scaled using the preset scale combination information of the interpolation model to be trained. Then, the optical flow residuals of the first scaled sample and the second scaled sample are predicted by each model in the interpolation model to be trained. This allows the interpolation model to learn video frames at different scales, thereby better fitting the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample. This improves the prediction accuracy of the pre-trained interpolation model for optical flow residuals.

[0086] In one embodiment, the frame interpolation model to be trained further includes an optical flow residual correction network. Step S202, which obtains the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample based on the first video frame sample, the second video frame sample, the actual intermediate frame sample between the first and second video frame samples, the first target optical flow, and the second target optical flow, specifically includes the following: inputting the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow into the optical flow residual correction network to obtain the third target optical flow residual corresponding to the first target optical flow and the fourth target optical flow residual corresponding to the second target optical flow; updating the first target optical flow and the second target optical flow accordingly based on the third and fourth target optical flow residuals to obtain the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample.

[0087] Specifically, the server inputs the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow into the optical flow residual correction network to predict the optical flow residual, thereby obtaining the third target optical flow residual and the fourth target optical flow residual; the third target optical flow residual is added to the first target optical flow to obtain the third target optical flow corresponding to the first video frame sample; the fourth target optical flow residual is added to the second target optical flow to obtain the fourth target optical flow corresponding to the second video frame sample.

[0088] In this embodiment, by inputting actual intermediate frame samples into the optical flow residual correction network, the optical flow residual correction network can learn knowledge that the first optical flow residual prediction model and the second optical flow residual prediction model have not learned. Subsequently, the third target optical flow and the fourth target optical flow output by the optical flow residual correction network can be used to construct a loss function to guide the learning of the first optical flow residual prediction model and the second optical flow residual prediction model, so as to improve the prediction accuracy of the pre-trained frame interpolation model for optical flow residuals.

[0089] In one embodiment, step S203 above, which obtains the loss function of the frame interpolation model to be trained based on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow, specifically includes the following: generating a first synthetic intermediate frame based on the first video frame sample, the second video frame sample, the first target optical flow, and the second target optical flow; generating a second synthetic intermediate frame based on the first video frame sample, the second video frame sample, the third target optical flow, and the fourth target optical flow; and obtaining the loss function of the frame interpolation model to be trained based on the first synthetic intermediate frame, the second synthetic intermediate frame, and the actual intermediate frame sample.

[0090] Specifically, the server deforms the first video frame sample and the second video frame sample according to the first target optical flow and the second target optical flow, respectively, to obtain deformed first video frame samples and deformed second video frame samples. The first video frame sample, the second video frame sample, the deformed first video frame sample, and the deformed second video frame sample are input into the intermediate frame fusion model to obtain the first synthesized intermediate frame between the first video frame sample and the second video frame sample output by the intermediate frame fusion model. The server deforms the first video frame sample and the second video frame sample again according to the third target optical flow and the fourth target optical flow, respectively, to obtain the second deformed first video frame sample and the second video frame sample. The first video frame sample, the second video frame sample, the second deformed first video frame sample, and the second deformed second video frame sample are input into the intermediate frame fusion model to obtain the second synthesized intermediate frame between the first video frame sample and the second video frame sample output by the intermediate frame fusion model. Based on the first synthesized intermediate frame, the second synthesized intermediate frame, the actual intermediate frame sample, and the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow, the loss function of the frame interpolation model to be trained is obtained.

[0091] In this embodiment, a first synthetic intermediate frame is synthesized using the first and second target optical flows output by the second optical flow residual prediction model in the frame interpolation model to be trained, as well as the first and second video frame samples. A second synthetic intermediate frame is synthesized using the third and fourth target optical flows output by the optical flow residual correction network in the frame interpolation model to be trained, as well as the first and second video frame samples. A loss function is calculated based on the first and second synthetic intermediate frames and the actual intermediate frame samples. Then, the model parameters of each model in the frame interpolation model to be trained are updated using gradient backpropagation to achieve reasonable training of the frame interpolation model to be trained, thereby obtaining a pre-trained frame interpolation model.

[0092] In one embodiment, the loss function of the frame interpolation model to be trained is obtained based on the first synthesized intermediate frame, the second synthesized intermediate frame, and the actual intermediate frame samples. Specifically, this includes: performing a Laplacian transform on the first synthesized intermediate frame and the actual intermediate frame samples to obtain a first Laplacian result; performing a Laplacian transform on the second synthesized intermediate frame and the actual intermediate frame samples to obtain a second Laplacian result; performing regularization on the first target optical flow and the third target optical flow to obtain a first regularization result; performing regularization on the second target optical flow and the fourth target optical flow to obtain a second regularization result; and obtaining the loss function of the frame interpolation model to be trained based on the first Laplacian result, the second Laplacian result, the first regularization result, and the second regularization result.

[0093] Specifically, the first Laplace result is labeled as Lap1, and the process of obtaining the first Laplace result can be represented by the following formula:

[0094]

[0095] in, G represents the first synthesized intermediate frame. t This represents the actual intermediate frame sample, and the lap() function represents the Laplace transform processing of the parameters in the lap() function.

[0096] The second Laplace result is labeled as Lap2. The process of obtaining the second Laplace result can be represented by the following formula:

[0097]

[0098] in, This indicates the second synthesized intermediate frame.

[0099] The first regularization result is labeled as L11. The process of obtaining the first regularization result can be represented by the following formula:

[0100] L11=L1(F′ t0 F t0 )

[0101] Among them, F' t0 F represents the optical flow of the third target. t0 The L1() function represents the first target optical flow, and the L1() function represents the regularization of the parameters in the L1() function.

[0102] The second regularization result is labeled as L12. The process of obtaining the second regularization result can be represented by the following formula:

[0103] L12=L1(F′ t1 F t1 )

[0104] Among them, F' t1 F represents the optical flow of the fourth target. t1 This represents the optical flow of the second target.

[0105] The server adds the first Laplacian result, the second Laplacian result, the first regularization result, and the second regularization result to obtain the loss function of the frame interpolation model to be trained. This loss function is denoted as Loss and can be expressed by the following formula:

[0106]

[0107] In this embodiment, the model parameters of each model in the frame interpolation model to be trained are updated by using a loss function and gradient backpropagation, which enables reasonable training of the frame interpolation model to be trained, thereby obtaining a pre-trained frame interpolation model.

[0108] In one embodiment, the pre-trained scale combination prediction model is trained as follows: First video frame samples and second video frame samples are input into a pre-trained interpolation model to obtain a first intermediate frame sample between the first and second video frame samples; the first video frame samples, the second video frame samples, and the first intermediate frame sample are input into the scale combination prediction model to be trained to obtain training scale combination information for the pre-trained interpolation model; the current scale combination information of the pre-trained interpolation model is adjusted according to the training scale combination information to obtain an adjusted interpolation model; the first video frame samples and second video frame samples are input into the adjusted interpolation model to obtain a second intermediate frame sample between the first and second video frame samples; the scale combination prediction model to be trained is corrected according to the peak signal-to-noise ratio or the learned perceptual image patch similarity between the first and second intermediate frame samples to obtain the pre-trained scale combination prediction model.

[0109] The pre-trained scale combination prediction model can be a trained reinforcement learning model, such as the Advantage Actor Critic (A2C) algorithm.

[0110] Specifically, the server inputs the first video frame sample and the second video frame sample into a pre-trained interpolation model to obtain the first optical flow corresponding to the first video frame sample and the second optical flow corresponding to the second video frame sample, which are output by another second optical flow residual prediction model in the pre-trained interpolation model. The first video frame sample, the second video frame sample, the first optical flow, and the second optical flow are input into an intermediate frame fusion model to obtain the first intermediate frame sample between the first video frame sample and the second video frame sample. The first video frame sample, the second video frame sample, and the first intermediate frame sample are input into a scale combination prediction model to be trained to obtain training scale combination information for the pre-trained interpolation model. Based on the training scale combination information, the current scale combination information in the pre-trained interpolation model is adjusted to obtain the adjusted interpolation model. The first video frame sample and the second video frame sample are input into the adjusted interpolation model to obtain the second intermediate frame sample between the first video frame sample and the second video frame sample.

[0111] A first peak signal-to-noise ratio (PSNR) is calculated based on the first intermediate frame sample and the real intermediate frame sample; a second PSNR is calculated based on the second intermediate frame sample and the real intermediate frame sample; the loss value is obtained based on the difference between the first PSNR and the second PSNR; or, a first learned perceptual image patch similarity is calculated based on the first intermediate frame sample and the real intermediate frame sample; a second learned perceptual image patch similarity is calculated based on the second intermediate frame sample and the real intermediate frame sample; the loss value is obtained based on the difference between the first learned perceptual image patch similarity and the second learned perceptual image patch similarity. Wherein, the first peak signal-to-noise ratio (PSNR) refers to the peak signal-to-noise ratio (PSNR) calculated based on the first intermediate frame sample and the real intermediate frame sample, and the second peak signal-to-noise ratio refers to the peak signal-to-noise ratio calculated based on the second intermediate frame sample and the real intermediate frame sample; the first learned perceptual image patch similarity (LPIPS) refers to the learned perceptual image patch similarity (LPIPS) calculated based on the first intermediate frame sample and the real intermediate frame sample, and the second learned perceptual image patch similarity (LPIPS) refers to the learned perceptual image patch similarity calculated based on the second intermediate frame sample and the real intermediate frame sample.

[0112] The server pre-sets a preset training quantity condition. When the training quantity is detected to meet the preset condition, the server updates the gradient of the scale combination prediction model to be trained based on the loss value. For example, the scale combination prediction model to be trained can use multiple threads to process multiple video frames in parallel. The server pre-sets the batch size to 1. When the training quantity meets the upper limit of the batch size, the loss values ​​obtained from each thread are merged in the 0 dimension to obtain the merged loss value. The merged loss value is then input into the scale combination prediction model to be trained for unified gradient updates.

[0113] In this embodiment, because the first and second video frame samples were scaled during the training process of the interpolation model to be trained, the size of the first and second video frame samples is mostly 224*224 and the motion amplitude is not large. This makes it difficult for the pre-trained interpolation model to fit intermediate frames with large motion amplitudes. Therefore, in this embodiment, the scale combination prediction model to be trained is trained to obtain a pre-trained scale combination prediction model. The current scale combination information of the pre-trained interpolation model is adjusted by the pre-trained scale combination prediction model, so that the pre-trained interpolation model can fit the intermediate frames well in both cases of large and small motion amplitudes of video frames, thereby improving the quality of the generated target intermediate frames.

[0114] In one embodiment, step S101, which involves inputting the first video frame and the second video frame into a pre-trained frame interpolation model to obtain an intermediate frame between the first video frame and the second video frame, specifically includes the following: scaling the first video frame and the second video frame according to the current scale combination information of the optical flow residual prediction network to obtain a scaled first video frame and a scaled second video frame; inputting the scaled first video frame and the scaled second video frame into the pre-trained frame interpolation model to obtain the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame; and obtaining the intermediate frame between the first video frame and the second video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow.

[0115] The second optical flow residual prediction model contains at least two optical flow residual prediction models. These two models have different model parameters but the same model structure. For example, these two optical flow residual prediction models can be convolutional neural networks with the same network structure, but they use different hyperparameters.

[0116] Specifically, when the server acquires the first video frame and the second video frame in the target video, it can scale the first video frame and the second video frame according to the first current scale in the current scale combination information of the optical flow residual prediction network to obtain the first scaled video frame and the second scaled video frame respectively. Then, the first scaled video frame and the second scaled video frame are input into the first optical flow residual prediction model in the pre-trained frame interpolation model to obtain the first initial optical flow residual corresponding to the first scaled video frame and the second initial optical flow residual corresponding to the second scaled video frame. The first initial optical flow residual is used as the first initial optical flow and the second initial optical flow residual is used as the second initial optical flow.

[0117] Based on the second current scale in the current scale combination information, the first video frame and the second video frame are scaled again to obtain the third scaled video frame and the fourth scaled video frame. The third scaled video frame, the fourth scaled video frame, the first initial optical flow and the second initial optical flow are input into a second optical flow residual prediction model in a pre-trained frame interpolation model to obtain the first updated optical flow residual of the first initial optical flow and the second updated optical flow residual of the second initial optical flow. The first initial optical flow and the first updated optical flow residual are added to obtain the first updated optical flow. The second initial optical flow and the second updated optical flow residual are added to obtain the second updated optical flow.

[0118] Based on the third current scale, the first video frame is scaled to obtain the fifth scaled video frame; based on the third current scale, the second video frame is scaled to obtain the sixth scaled video frame; the fifth scaled video frame, the sixth scaled video frame, the first updated optical flow, and the second updated optical flow are input into another second optical flow residual prediction model in the frame interpolation model to be trained to obtain the first optical flow residual and the second optical flow residual; the first optical flow residual is added to the first updated optical flow to obtain the first optical flow; the second optical flow residual is added to the second updated optical flow to obtain the second optical flow. The third current scale can be set to 0, meaning that no scaling is performed on the first and second video frames.

[0119] The first video frame, the second video frame, the first optical flow, and the second optical flow are input into the intermediate frame fusion model to obtain the intermediate frame between the first video frame and the second video frame.

[0120] In this embodiment, the first and second video frames are scaled using the current scale combination information to obtain scaled first and second video frames. This allows for reasonable adjustment of the scales of the first and second video frames. The scaled first and second video frames are then input into a pre-trained frame interpolation model to obtain the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame. Based on the first and second video frames, the first and second optical flows, an intermediate frame between the first and second video frames is obtained, achieving reasonable generation of the intermediate frame. Furthermore, based on the predicted scale combination information provided by the pre-trained scale combination prediction model, the current scale combination information of the pre-trained frame interpolation model can be continuously updated, greatly improving the quality of the generated intermediate frames.

[0121] In one embodiment, step S102 above, the downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame, specifically includes the following: performing color channel splicing processing on the first video frame, the intermediate frame, and the second video frame to obtain a spliced ​​video frame; and performing downsampling processing on the spliced ​​video frame to obtain a downsampled video frame.

[0122] The number of channels in the spliced ​​video frames is equal to the sum of the number of channels in the first video frame, the intermediate frames, and the second video frame.

[0123] Specifically, the server performs concatenation processing on the first video frame, intermediate frames, and second video frame in the red, green, and blue color channels to obtain a concatenated video frame; the concatenated video frame is then downsampled to obtain a downsampled video frame. The downsampling scale includes, but is not limited to, 1 / 8, 1 / 16, and 1 / 24.

[0124] For example, the spliced ​​video frame has 9 channels. The server downsamples the spliced ​​video frame to 1 / 16 of the spliced ​​video frame, resulting in a downsampled video frame with a length and width that are 1 / 4 of the length of the spliced ​​video frame, but with 9 channels.

[0125] In this embodiment, downsampling is performed on the stitched video frames to obtain downsampled video frames, thereby reducing the computational load of the pre-trained scale combination prediction model on the downsampled video frames, which in turn reduces the prediction time of the pre-trained scale combination prediction model and improves the efficiency of obtaining prediction scale combination information.

[0126] In one embodiment, the video frame interpolation method further includes: evaluating intermediate frames using a pre-trained frame interpolation model to obtain evaluation results for the intermediate frames; confirming that a preset termination condition is met if the evaluation results meet a preset evaluation threshold; or confirming that a preset termination condition is met if the number of times intermediate frames are acquired reaches a preset number.

[0127] Specifically, the server can evaluate the input intermediate frames using a pre-trained frame interpolation model to obtain the evaluation results. For example, it can compare the spatiotemporal coherence of the first video frame, intermediate frames, and second video frames in the form of a network black box. If the evaluation result is greater than a preset evaluation threshold, a preset termination condition is confirmed. The server can also confirm that the preset termination condition is met when the number of intermediate frames obtained is greater than or equal to a preset number. When the server confirms that the preset termination condition is met, it outputs the currently obtained intermediate frame as the target video frame between the first and second video frames.

[0128] In one embodiment, such as Figure 4 As shown, another video frame interpolation method is provided. Taking the application of this method to a server as an example, the following steps are included:

[0129] Step S401: Based on the current scale combination information of the optical flow residual prediction network, scale the first video frame and the second video frame to obtain scaled first video frame and scaled second video frame.

[0130] Step S402: Input the scaled first video frame and the scaled second video frame into the pre-trained frame interpolation model to obtain the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame.

[0131] Step S403: Based on the first video frame, the second video frame, the first optical flow, and the second optical flow, obtain the intermediate frame between the first video frame and the second video frame.

[0132] Step S404: Perform color channel splicing on the first video frame, the intermediate frame, and the second video frame to obtain a spliced ​​video frame; perform downsampling on the spliced ​​video frame to obtain a downsampled video frame.

[0133] Step S405: Input the downsampled video frames into the pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained frame interpolation model.

[0134] Step S406: Adjust the current scale combination information of the pre-trained frame interpolation model according to the predicted scale combination information to obtain the adjusted frame interpolation model; use the adjusted frame interpolation model as the corresponding pre-trained frame interpolation model, and jump to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until the preset termination condition is met.

[0135] Step S407: The intermediate frame that meets the preset termination condition is taken as the target video frame between the first video frame and the second video frame.

[0136] The above video frame interpolation method can achieve the following beneficial effects: inputting downsampled video frames into a pre-trained scale combination prediction model not only enables the acquisition of predicted scale combination information using video frames smaller than the first video frame, intermediate frame, and second video frame, but also reduces the computational load of the pre-trained scale combination prediction model and improves processing efficiency; by adjusting the current scale combination information in the pre-trained frame interpolation model through the pre-trained scale combination prediction model, the intermediate frame can be well fitted regardless of whether the video frame has a large or small motion amplitude, thereby improving the quality of the generated target intermediate frame.

[0137] To more clearly illustrate the video frame interpolation method provided in this disclosure, a specific embodiment will be used to describe the method in detail below. For example... Figure 5 As shown, in one embodiment, another video frame interpolation method is provided, which can be applied to a server and can be divided into a model training process and a frame interpolation process, specifically including the following:

[0138] The model training process includes: (1) Image preprocessing: Extract three consecutive video frames from the sample video and label them as A, B and C. At the same time, divide the three frames A, B and C by 255 and normalize them to between 0 and 1. The red, green and blue channels of the normalized A, B and C can be randomly reversed. The video frames A, B and C can be randomly reversed vertically and horizontally. A portion can also be randomly cut to obtain the first video frame sample I0 and the actual intermediate frame sample G. t And the second video frame sample I1.

[0139] (2) Train the frame interpolation model to be trained: Figure 6 The diagram shows the structure of the frame interpolation model to be trained, as follows: Figure 6 As shown, the frame interpolation model to be trained can consist of a first optical flow residual prediction model, two second optical flow residual prediction models, an optical flow residual correction network, and an intermediate frame fusion model. The first optical flow residual prediction model, the two second optical flow residual prediction models, and the optical flow residual correction network can be a combination of multi-layer convolutional neural networks. Therefore, the frame interpolation model to be trained can also be implemented through a convolutional neural network, and training the frame interpolation model to be trained can be regarded as a regression learning process.

[0140] Specifically, the first preset scale can be 1 / 4. Then, I0 and I1 are reduced by one-quarter and input into the first optical flow residual prediction model to obtain the first initial optical flow residual ΔFL of I0. t0 The second initial optical flow residual ΔFL of I1 t1 Because the server initially does not know the optical flow FL of I0 and I1. t0 and FL t1 , then FL t0 and FL t1 All can be set to 0, thus the first initial optical flow FL' t0 =△FL t0 Second initial optical flow FL' t1 =△FL t1 The second preset scale can be 1 / 2. After reducing I0 and I1 by half, input them into the second optical flow residual prediction model, and then input FL'... t0 and FL' t1 The input is given to the second optical flow residual prediction model to obtain the first updated optical flow residual ΔFL'. t0 Second update optical flow residual ΔFL' t1 Then the first update optical flow is FL. t0 =△FL' t0 +FL' t0 The second update optical flow is FL. t1 =△FL' t1 +FL' t1 The third preset scale can be 0, in which case there is no need to scale I0 and I1, and I0, I1, and FL will be scaled. t0 and FL” t1 Input the second optical flow residual prediction model to obtain the first target optical flow residual ΔFL”. t0 Second target optical flow residual ΔFL” t1 Then the optical flow of the first target is F. t0 =△FL” t0 +FL” t0 The optical flow of the second target is F t1 =△FL” t1+FL” t1 Set I0, I1, and F t0 and F t1 The input is fed into the intermediate frame fusion model to obtain the first synthesized intermediate frame. Set I0, I1, F t0 F t1 and G t The input is fed into the optical flow residual correction network to obtain ΔF'. t0 and △F' t1 Then the optical flow of the third target is F' t0 =△F' t0 +F t0 The optical flow of the fourth target is F' t1 =△F' t1 +F t1 Set I0, I1, and F' t0 and F' t1 The input is fed into the intermediate frame fusion model to obtain the second synthesized intermediate frame. The loss function Loss of the frame interpolation model to be trained can be obtained using the following formula:

[0141]

[0142] By using the loss function Loss, the model parameters of each model in the frame interpolation model to be trained are updated through gradient backpropagation, thus obtaining the pre-trained frame interpolation model.

[0143] (3) Training the scale combination prediction model to be trained: In step (2), when training the interpolation model to be trained, the first video frame sample I0 and the second video frame sample I1 are gradually reduced by 1 / 4, 1 / 2 and not reduced video frame samples to train the interpolation model to be trained. Since the video frame samples are generally 224*224 in size and the motion amplitude is not large, it is difficult for the pre-trained interpolation model to fit the intermediate frame with a large motion amplitude. Therefore, the scale combination prediction model is used to adjust the current scale combination information in the pre-trained interpolation model. Figure 7 This is a schematic diagram of the structure of a pre-trained scale combination prediction model, such as... Figure 7 As shown, I0 and the intermediate frame output by the pre-trained frame interpolation model for the i-th time are... I1 is input into the scale combination prediction model to be trained to obtain the predicted scale combination information; based on this predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model and the intermediate frame output by the adjusted frame interpolation model at the (i+1)th iteration. Based on the intermediate frame of the (i+1)th output and actual intermediate frame sample G t , obtain PSNR i+1(or LPIPS) i+1 Then, based on the intermediate frame of the i-th output... and actual intermediate frame sample G t , obtain PSNR i (or LPIPS) i ), via PSNR i Subtract PSNR i+1 (or LPIPS) i Subtract LPIPS i+1 The loss value is obtained, and the loss value is used to correct the scale combination prediction model to be trained, so as to obtain the pre-trained scale combination prediction model.

[0144] (4) Generating model files: After training of the frame interpolation model and the scale combination prediction model to be trained is completed, and the pre-trained frame interpolation model and the pre-trained scale combination prediction model are obtained, the necessary information of the pre-trained frame interpolation model and the pre-trained scale combination prediction model is saved to generate model files. The necessary information includes the model parameters of the pre-trained frame interpolation model and the pre-trained scale combination prediction model.

[0145] Figure 8 This is a schematic diagram illustrating the principle of another video frame interpolation method in this embodiment, as shown below. Figure 8 As shown, the frame interpolation process includes: processing the first video frame M0 and M1 and the second video frame using a pre-trained frame interpolation model to obtain an intermediate frame between the first and second video frames; wherein, the current scale combination information in the pre-trained frame interpolation model is 1 / 4, 1 / 2, and 1x (1x means the original video frame is entered). M0, M1 and M1 are concatenated on the RGB channels, then downsampled to 1 / 16th of the original video frame, resulting in a state variable S with 9 channels but a width and height reduced to 1 / 4 of the original video frame. S is input into a pre-trained scale combination prediction model to obtain predicted scale combination information. This predicted scale combination information is a combination of various scaling factors, including 1 / 16, 1 / 8, 1 / 4, 1 / 2, and 1x. The pre-trained scale combination prediction model selects several of these factors for combination to obtain the predicted scale combination information. This predicted scale combination information is then re-input into the first optical flow residual prediction model and two second optical flow residual prediction models in the pre-trained frame interpolation model to obtain the adjusted frame interpolation model. The new intermediate frame output by the adjusted frame interpolation model is then obtained. Then M0, The first video frame and the second video frame are concatenated on the RGB channels, then downsampled to 1 / 16th of the original video frame. The resulting state S is then input into a pre-trained scale combination prediction model to obtain new predicted scale combination information. This process is repeated until the pre-trained scale combination prediction model confirms that the input intermediate frame meets the preset video frame conditions, or the preset maximum number of iterations is reached. At this point, the intermediate frame obtained in the current iteration is output as the target intermediate frame. The first video frame, the target intermediate frame, and the second video frame are considered the optimal interpolation combination between the first and second video frames.

[0146] In this embodiment, the current scale combination information in the pre-trained frame interpolation model is recombined by selecting different scales through a pre-trained scale combination prediction model. This avoids the problem of losing details when using large-scale scaling for small movements and failing to fit when using small-scale scaling for large movements. At the same time, the pre-trained scale combination prediction model only needs 1 / 16 of the size of the original video frame to accurately output the predicted scale combination information and generate the best intermediate frame within a given loop, thus obtaining the best frame interpolation combination.

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides a video frame interpolation apparatus for implementing the video frame interpolation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video frame interpolation apparatus embodiments provided below can be found in the limitations of the video frame interpolation method described above, and will not be repeated here.

[0149] In one embodiment, such as Figure 9 As shown, a video frame interpolation device 900 is provided, including: an intermediate frame acquisition module 901, a scale combination prediction module 902, an interpolation model adjustment module 903, and a video frame determination module 904, wherein:

[0150] The intermediate frame acquisition module 901 is used to input the first video frame and the second video frame into a pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame.

[0151] The scale combination prediction module 902 is used to input the downsampled video frames obtained based on the first video frame, the intermediate frame and the second video frame into the pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained frame interpolation model.

[0152] The frame interpolation model adjustment module 903 is used to adjust the current scale combination information of the pre-trained frame interpolation model according to the prediction scale combination information to obtain the adjusted frame interpolation model.

[0153] The video frame determination module 904 is used to take the adjusted interpolation model as the pre-trained interpolation model, and jump to the step of inputting the first video frame and the second video frame into the pre-trained interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is taken as the target video frame between the first video frame and the second video frame.

[0154] In one embodiment, the video frame interpolation device 900 further includes a frame interpolation model training module, configured to input a first video frame sample and a second video frame sample into the frame interpolation model to be trained, to obtain a first target optical flow corresponding to the first video frame sample and a second target optical flow corresponding to the second video frame sample; to obtain a third target optical flow corresponding to the first video frame sample and a fourth target optical flow corresponding to the second video frame sample based on the first video frame sample, the second video frame sample, the actual intermediate frame sample between the first and second video frame samples, the first target optical flow, and the second target optical flow; to obtain a loss function of the frame interpolation model to be trained based on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow; and to perform iterative training on the frame interpolation model to be trained based on the loss function to obtain a pre-trained frame interpolation model.

[0155] In one embodiment, the video frame interpolation device 900 further includes a first optical flow acquisition module, configured to scale a first video frame sample and a second video frame sample according to a first preset scale in the preset scale combination information of the frame interpolation model to be trained, to obtain a first scaled sample and a second scaled sample; input the first scaled sample and the second scaled sample into a first optical flow residual prediction model in the frame interpolation model to be trained, to obtain a first initial optical flow residual corresponding to the first scaled sample and a second initial optical flow residual corresponding to the second scaled sample, and use the first initial optical flow residual as the first initial optical flow and the second initial optical flow residual as the second initial optical flow; scale the first video frame sample and the second video frame sample according to the second preset scale in the preset scale combination information, to obtain a third scaled sample and a fourth scaled sample; input the third scaled sample, the fourth scaled sample, the first initial optical flow, and the second initial optical flow into a second optical flow prediction model in the frame interpolation model to be trained. The optical flow residual prediction model obtains a first updated optical flow residual and a second updated optical flow residual of the first initial optical flow and the second initial optical flow. The first updated optical flow residual is used to update the first initial optical flow to obtain a first updated optical flow. The second updated optical flow residual is used to update the second initial optical flow to obtain a second updated optical flow. According to the third preset scale in the preset scale combination information, the first video frame sample and the second video frame sample are scaled to obtain a fifth scaled sample and a sixth scaled sample. The fifth scaled sample, the sixth scaled sample, the first updated optical flow, and the second updated optical flow are input into the second optical flow residual prediction model to obtain a first target optical flow residual of the first updated optical flow and a second target optical flow residual of the second updated optical flow. According to the first target optical flow residual and the second target optical flow residual, the first updated optical flow and the second updated optical flow are updated accordingly to obtain the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample.

[0156] In one embodiment, the frame interpolation model to be trained further includes an optical flow residual correction network; the video frame interpolation device 900 further includes a second optical flow acquisition module, used to input the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow into the optical flow residual correction network to obtain the third target optical flow residual corresponding to the first target optical flow and the fourth target optical flow residual corresponding to the second target optical flow; based on the third target optical flow residual and the fourth target optical flow residual, the first target optical flow and the second target optical flow are updated accordingly to obtain the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample.

[0157] In one embodiment, the video frame interpolation device 900 further includes a loss function acquisition module, configured to generate a first synthetic intermediate frame based on a first video frame sample, a second video frame sample, a first target optical flow, and a second target optical flow; generate a second synthetic intermediate frame based on the first video frame sample, the second video frame sample, a third target optical flow, and a fourth target optical flow; and obtain the loss function of the frame interpolation model to be trained based on the first synthetic intermediate frame, the second synthetic intermediate frame, and actual intermediate frame samples.

[0158] In one embodiment, the video frame interpolation device 900 further includes a loss function determination module, which is used to perform Laplacian transform processing on the first synthesized intermediate frame and the actual intermediate frame samples to obtain a first Laplacian result; perform Laplacian transform processing on the second synthesized intermediate frame and the actual intermediate frame samples to obtain a second Laplacian result; perform regularization processing on the first target optical flow and the third target optical flow to obtain a first regularization result; perform regularization processing on the second target optical flow and the fourth target optical flow to obtain a second regularization result; and obtain the loss function of the frame interpolation model to be trained based on the first Laplacian result, the second Laplacian result, the first regularization result, and the second regularization result.

[0159] In one embodiment, the video interpolation device 900 further includes a scale combination prediction model training module, configured to: input a first video frame sample and a second video frame sample into a pre-trained interpolation model to obtain a first intermediate frame sample between the first and second video frame samples; input the first video frame sample, the second video frame sample, and the first intermediate frame sample into a scale combination prediction model to be trained to obtain training scale combination information for the pre-trained interpolation model; adjust the current scale combination information of the pre-trained interpolation model according to the training scale combination information to obtain an adjusted interpolation model; input the first and second video frame samples into the adjusted interpolation model to obtain a second intermediate frame sample between the first and second video frame samples; and correct the scale combination prediction model to be trained according to the peak signal-to-noise ratio or the learned perceptual image patch similarity between the first and second intermediate frame samples to obtain a pre-trained scale combination prediction model.

[0160] In one embodiment, the intermediate frame acquisition module 901 is further configured to scale the first video frame and the second video frame according to the current scale combination information of the optical flow residual prediction network to obtain the scaled first video frame and the scaled second video frame; input the scaled first video frame and the scaled second video frame into a pre-trained frame interpolation model to obtain the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame; and obtain the intermediate frame between the first video frame and the second video frame according to the first video frame, the second video frame, the first optical flow and the second optical flow.

[0161] In one embodiment, the scale combination prediction module 902 is further configured to perform color channel splicing processing on the first video frame, the intermediate frame, and the second video frame to obtain a spliced ​​video frame; and to perform downsampling processing on the spliced ​​video frame to obtain a downsampled video frame.

[0162] In one embodiment, the video frame interpolation device 900 further includes a termination condition confirmation module, which is used to evaluate the intermediate frame through a pre-trained frame interpolation model to obtain the evaluation result of the intermediate frame; if the evaluation result meets a preset evaluation threshold, confirm that the preset termination condition is met; or, if the number of times the intermediate frame is acquired reaches a preset number, confirm that the preset termination condition is met.

[0163] Each module in the aforementioned video frame interpolation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0164] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores video frame data, such as first video frames, second video frames, and intermediate frames. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a video frame interpolation method.

[0165] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0166] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0167] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0168] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video frame interpolation method, characterized in that, The method includes: The first video frame and the second video frame are input into a pre-trained frame interpolation model to obtain an intermediate frame between the first video frame and the second video frame; wherein, the pre-trained frame interpolation model is obtained by iteratively training the frame interpolation model to be trained based on the first video frame sample, the second video frame sample, the first target optical flow corresponding to the first video frame sample, and the second target optical flow corresponding to the second video frame sample; The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame is input into a pre-trained scale combination prediction model to obtain the prediction scale combination information of the pre-trained frame interpolation model. Based on the predicted scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model. The adjusted frame interpolation model is used as the pre-trained frame interpolation model, and the process jumps to the step of inputting the first video frame and the second video frame into the pre-trained frame interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is used as the target video frame between the first video frame and the second video frame. The first target optical flow and the second target optical flow are obtained in the following manner: According to the first preset scale in the preset scale combination information of the frame interpolation model to be trained, the first video frame sample and the second video frame sample are scaled to obtain scaled samples under the first preset scale; each scaled sample is input into the first optical flow residual prediction model in the frame interpolation model to be trained to obtain the initial optical flow residual corresponding to each scaled sample, and each initial optical flow residual is used as the corresponding initial optical flow. Obtain the second preset scale from the preset scale combination information, and scale the first video frame sample and the second video frame sample again based on the second preset scale to obtain the scaled sample under the second preset scale; input the scaled sample under the second preset scale and each of the initial optical flows into the second optical flow residual prediction model in the frame interpolation model to be trained to obtain the updated optical flow residual of each of the initial optical flows, and update each of the initial optical flows based on each of the updated optical flow residuals to obtain each updated optical flow; Obtain the third preset scale from the preset scale combination information, repeat the above steps to obtain each updated optical flow based on the third preset scale, and take each finally obtained updated optical flow as the first target optical flow and the second target optical flow, respectively.

2. The method according to claim 1, characterized in that, The pre-trained frame interpolation model is trained in the following manner: The first video frame sample and the second video frame sample are input into the frame interpolation model to be trained to obtain the first target optical flow corresponding to the first video frame sample and the second target optical flow corresponding to the second video frame sample. Based on the first video frame sample, the second video frame sample, the actual intermediate frame sample between the first video frame sample and the second video frame sample, the first target optical flow, and the second target optical flow, the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample are obtained. The loss function of the frame interpolation model to be trained is obtained based on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the third target optical flow, and the fourth target optical flow. The frame interpolation model to be trained is iteratively trained according to the loss function to obtain the pre-trained frame interpolation model.

3. The method according to claim 2, characterized in that, The frame interpolation model to be trained also includes an optical flow residual correction network; The step of obtaining the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample based on the first video frame sample, the second video frame sample, the actual intermediate frame sample between the first video frame sample and the second video frame sample, the first target optical flow, and the second target optical flow includes: The first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, and the second target optical flow are input into the optical flow residual correction network to obtain the third target optical flow residual corresponding to the first target optical flow and the fourth target optical flow residual corresponding to the second target optical flow. Based on the third target optical flow residual and the fourth target optical flow residual, the first target optical flow and the second target optical flow are updated accordingly to obtain the third target optical flow corresponding to the first video frame sample and the fourth target optical flow corresponding to the second video frame sample.

4. The method according to claim 2, characterized in that, The loss function for obtaining the frame interpolation model to be trained based on the first video frame sample, the second video frame sample, the actual intermediate frame sample, the first target optical flow, the second target optical flow, the three target optical flows, and the fourth target optical flow includes: A first synthesized intermediate frame is generated based on the first video frame sample, the second video frame sample, the first target optical flow, and the second target optical flow. A second synthesized intermediate frame is generated based on the first video frame sample, the second video frame sample, the third target optical flow, and the fourth target optical flow. The loss function of the frame interpolation model to be trained is obtained based on the first synthesized intermediate frame, the second synthesized intermediate frame, and the actual intermediate frame sample.

5. The method according to claim 4, characterized in that, The step of obtaining the loss function of the frame interpolation model to be trained based on the first synthesized intermediate frame, the second synthesized intermediate frame, and the actual intermediate frame samples includes: The first synthesized intermediate frame and the actual intermediate frame sample are subjected to Laplacian transform to obtain the first Laplacian result. The second synthesized intermediate frame and the actual intermediate frame sample are subjected to Laplacian transform to obtain the second Laplacian result. The first target optical flow and the third target optical flow are regularized to obtain the first regularization result; The second target optical flow and the fourth target optical flow are regularized to obtain a second regularization result; Based on the first Laplacian result, the second Laplacian result, the first regularization result, and the second regularization result, the loss function of the frame interpolation model to be trained is obtained.

6. The method according to claim 1, characterized in that, The pre-trained scale combination prediction model is trained in the following manner: The first video frame sample and the second video frame sample are input into the pre-trained frame interpolation model to obtain the first intermediate frame sample between the first video frame sample and the second video frame sample. The first video frame sample, the second video frame sample, and the first intermediate frame sample are input into the scale combination prediction model to be trained to obtain the training scale combination information for the pre-trained frame interpolation model. Based on the training scale combination information, the current scale combination information of the pre-trained frame interpolation model is adjusted to obtain the adjusted frame interpolation model. The first video frame sample and the second video frame sample are input into the adjusted frame interpolation model to obtain the second intermediate frame sample between the first video frame sample and the second video frame sample; The scale combination prediction model to be trained is modified based on the peak signal-to-noise ratio or the learned perceptual image patch similarity between the first intermediate frame sample and the second intermediate frame sample to obtain the pre-trained scale combination prediction model; the learned perceptual image patch similarity is calculated based on the first intermediate frame sample and the second intermediate frame sample.

7. The method according to claim 1, characterized in that, The step of inputting the first video frame and the second video frame into a pre-trained frame interpolation model to obtain an intermediate frame between the first video frame and the second video frame includes: Based on the current scale combination information of the optical flow residual prediction network, the first video frame and the second video frame are scaled to obtain the scaled first video frame and the scaled second video frame. The scaled first video frame and the scaled second video frame are input into the pre-trained frame interpolation model to obtain the first optical flow corresponding to the first video frame and the second optical flow corresponding to the second video frame. An intermediate frame between the first video frame and the second video frame is obtained based on the first video frame, the second video frame, the first optical flow, and the second optical flow.

8. The method according to claim 1, characterized in that, The downsampled video frame obtained based on the first video frame, the intermediate frame, and the second video frame includes: The first video frame, the intermediate frame, and the second video frame are subjected to color channel splicing to obtain a spliced ​​video frame. The spliced ​​video frames are downsampled to obtain the downsampled video frames.

9. The method according to claim 1, characterized in that, The method further includes: The intermediate frames are evaluated using the pre-trained frame interpolation model to obtain evaluation results; if the evaluation results meet a preset evaluation threshold, the preset termination condition is confirmed to be met; or... If the number of times intermediate frames are acquired reaches a preset number, the preset termination condition is confirmed to be met.

10. A video frame interpolation device, characterized in that, The device includes: The intermediate frame acquisition module is used to input the first video frame and the second video frame into a pre-trained frame interpolation model to obtain an intermediate frame between the first video frame and the second video frame; wherein, the pre-trained frame interpolation model is obtained by iteratively training the frame interpolation model to be trained based on the first video frame sample, the second video frame sample, the first target optical flow corresponding to the first video frame sample, and the second target optical flow corresponding to the second video frame sample; The scale combination prediction module is used to input the downsampled video frame obtained based on the first video frame, the intermediate frame and the second video frame into the pre-trained scale combination prediction model to obtain the predicted scale combination information of the pre-trained frame interpolation model. The frame interpolation model adjustment module is used to adjust the current scale combination information of the pre-trained frame interpolation model according to the prediction scale combination information to obtain the adjusted frame interpolation model. The video frame determination module is used to take the adjusted interpolation model as the pre-trained interpolation model, and jump to the step of inputting the first video frame and the second video frame into the pre-trained interpolation model to obtain the intermediate frame between the first video frame and the second video frame, until a preset termination condition is met, and the intermediate frame corresponding to the preset termination condition is taken as the target video frame between the first video frame and the second video frame. The device further includes a first optical flow acquisition module, configured to scale the first video frame sample and the second video frame sample according to a first preset scale in the preset scale combination information of the interpolation model to be trained, to obtain scaled samples at the first preset scale; input each scaled sample into a first optical flow residual prediction model in the interpolation model to be trained, to obtain an initial optical flow residual corresponding to each scaled sample, and use each initial optical flow residual as the corresponding initial optical flow; acquire a second preset scale in the preset scale combination information, and scale the first video frame sample and the second video frame sample according to the second preset scale. The second video frame sample is scaled again to obtain a scaled sample at the second preset scale. The scaled sample at the second preset scale and each initial optical flow are input into the second optical flow residual prediction model in the frame interpolation model to be trained to obtain the updated optical flow residual of each initial optical flow. The initial optical flow is updated based on the updated optical flow residual to obtain each updated optical flow. The third preset scale in the preset scale combination information is obtained. The steps to obtain each updated optical flow are repeated based on the third preset scale. The finally obtained updated optical flows are used as the first target optical flow and the second target optical flow, respectively.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images