A method, system, device, medium, and product for film interpolation

Abstract

The application discloses a movie frame interpolation method, system, device, medium and product, relates to the field of movie frame interpolation, and comprises the following steps: using high frame rate video in an RGB color space as training data, training a video frame interpolation model, determining the trained video frame interpolation model, estimating the motion of input low frame rate video frames, and determining a frame interpolation result; inputting the low frame rate video frames and the frame interpolation result into a color correction module, correcting the frame interpolation result through a color space random disturbance mode, and determining a final frame interpolation result; the color correction module is a post-processing module of frame interpolation, is used for enhancing the video frame interpolation algorithm when the parameters of the trained video frame interpolation model remain unchanged, and randomly converting RGB data to different color spaces during training, so that the RGB data can correct the negative effects of different color space frame interpolation, and can ensure that there is no color deviation and other negative effects caused by color space during final playing.

Description

A method, system, device, medium, and product for movie frame interpolation. Technical Field

[0001] This application relates to the field of film frame interpolation, and in particular to a film frame interpolation method, system, device, medium, and product. Background Technology

[0002] Currently, most films are shot and produced at a frame rate of 24 frames per second (fps). However, due to advancements in display and processing devices, high frame rate films are gradually becoming a mainstream demand. Neural network-based video frame interpolation technology can overcome the limitations imposed by acquisition and hardware devices during film production through software algorithms, increasing the film's frame rate and providing viewers with a smoother, more immersive viewing experience. Although frame interpolation has undergone years of research and iteration in both academia and industry, the challenges faced by film frame interpolation are more complex and varied than those of traditional video frame interpolation, especially in supporting different color spaces.

[0003] Traditional video frame interpolation follows the principle of "training-test consistency" in terms of color space, meaning that the training and test data have the same color space to ensure that the interpolated result matches the color space of the input frame. Unlike traditional video frame interpolation algorithms, film uses a much more varied color space. Besides the common RGB color space, there are YUV, XYZ, and some film-specific color grading. Ignoring these color space differences during interpolation can lead to negative effects such as color shifts in the interpolated result due to color space mismatch. Summary of the Invention

[0004] The purpose of this application is to provide a movie frame interpolation method, system, device, medium, and product to solve the problem of negative effects such as color shift in the frame interpolation results caused by color space mismatch.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In a first aspect, this application provides a method for movie frame interpolation, comprising the following steps.

[0007] High frame rate videos in the RGB color space are used as training data to train the video frame interpolation model and determine the trained video frame interpolation model.

[0008] The trained video interpolation model is used to estimate the motion of the input low frame rate video frames to determine the interpolation result.

[0009] The low frame rate video frames and the interpolation results are input to the color correction module. The interpolation results are corrected through random perturbation of the color space to determine the final interpolation result. The color correction module is used to enhance the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged. This enhances the video interpolation algorithm to interpolate frames in different color spaces, resulting in interpolation results without negative effects. As a post-processing module for interpolation, the color correction module randomly converts the input RGB data to different color spaces during training, enabling the RGB data to correct for negative effects of interpolation in different color spaces. The RGB data refers to high frame rate video in the RGB color space, and the different color spaces include the XYZ color space and the YUV color space. The XYZ color space includes XYZ-E and XYZ-D65.

[0010] Secondly, this application provides a movie frame interpolation method system, including the following modules.

[0011] The training module is used to train the video frame interpolation model using high frame rate videos in the RGB color space as training data, and to determine the trained video frame interpolation model.

[0012] The interpolation result determination module is used to estimate the motion of the input low frame rate video frames using the trained video interpolation model and determine the interpolation result.

[0013] The correction module is used to input the low frame rate video frames and the interpolation results into the color correction module, and correct the interpolation results through random perturbation of the color space to determine the final interpolation result. The color correction module is used to enhance the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged, so that the enhanced video interpolation algorithm can interpolate frames in different color spaces and obtain interpolation results without negative effects. As a post-processing module for interpolation, the color correction module randomly converts the input RGB data to different color spaces during training, so that the RGB data can correct the negative effects of interpolation in different color spaces. The RGB data is high frame rate video in the RGB color space, and the different color spaces include XYZ color space and YUV color space. The XYZ color space includes XYZ-E and XYZ-D65.

[0014] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described movie frame interpolation method.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described movie frame interpolation method.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described movie frame interpolation method.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects:

[0018] This application is divided into two stages. The first stage is the video interpolation model training stage, which uses high frame rate videos in RGB color space as training data, so that the trained video interpolation model can stably interpolate frames for input videos of different types and specifications. The second stage integrates a color correction module, which corrects the interpolation results through random perturbation of the color space to ensure that there are no negative effects such as color shift due to color space during final playback, and determines the final interpolation result. Thus, without changing the interpolation capability of the video interpolation model, it can process input videos in different color spaces. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a schematic flowchart of a movie frame interpolation method provided in an embodiment of this application.

[0021] Figure 2 is a schematic diagram of the training phase of a video frame interpolation model provided in an embodiment of this application.

[0022] Figure 3 is a schematic diagram of a two-stage video frame interpolation training scheme provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The characteristics of film video color spaces present challenges for the development of film frame interpolation algorithms.

[0026] The interpolated video may require further post-processing, so the algorithm's input and output must be in the same color space. The input video cannot be converted to a uniform color space before interpolation.

[0027] Color spaces have different standards, and it's impossible to guarantee a reversible conversion between the input color space and a specific standard color space, such as reversible conversion between RGB and other color spaces. Therefore, the algorithm cannot maintain a stable input color space by performing frame interpolation after external conversion and then reversing the conversion.

[0028] There are numerous color space standards; taking the XYZ format as an example, there are different color formats such as XYZ-E and XYZ-D65. Especially to ensure the best presentation of film quality and content, algorithms may need to perform unpredictable and customized color conversions.

[0029] Based on this, as shown in Figure 1, this application provides a movie frame interpolation method, including the following steps.

[0030] S1: Use high frame rate videos in the RGB color space as training data to train the video frame interpolation model and determine the trained video frame interpolation model.

[0031] S2: Use the trained video interpolation model to estimate the motion of the input low frame rate video frames and determine the interpolation result.

[0032] S3: The low frame rate video frame and the interpolation result are input to the color correction module. The interpolation result is corrected by random perturbation of the color space to determine the final interpolation result. The color correction module is used to enhance the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged, so that the enhanced video interpolation algorithm can interpolate frames in different color spaces and obtain interpolation results without negative effects. As a post-processing module for interpolation, the color correction module randomly converts the input RGB data to different color spaces during training, so that the RGB data can correct the negative effects of interpolation in different color spaces. The RGB data is a high frame rate video in the RGB color space. Different color spaces include XYZ color space and YUV color space. The XYZ color space includes XYZ-E and XYZ-D65. The negative effects include color shift.

[0033] Previous video frame interpolation techniques followed the "training-test consistency" principle, meaning the color spaces of the training and test sets were completely identical. However, film frame interpolation, due to the video characteristics and production limitations of film, requires the same algorithm to support interpolation in different color spaces. Simply using existing video frame interpolation techniques directly for film frame interpolation can lead to negative effects such as color shift. To address this issue, this application proposes a color correction module that uses the input frame to correct the color of the interpolation result. Furthermore, the proposed color correction module can be easily and directly integrated with existing video frame interpolation schemes without affecting their ability to handle video motion or the interpolation effect.

[0034] In addition, this application proposes a two-stage video frame interpolation training scheme, which includes training data generation, training scheme for the video frame interpolation model, and training target loss. This helps the algorithm to quickly and stably complete the training of the color correction module while ensuring the effectiveness of the video frame interpolation model.

[0035] Among them, training data generation: During the training process, the color space of the input RGB data is randomly transformed and the transformation function is perturbed to model the input data in different color spaces and generate training data.

[0036] In one exemplary embodiment, S2 specifically includes the following steps.

[0037] S21: The motion of the input low frame rate video frame is estimated using the trained video interpolation model; the low frame rate video frame includes two frames.

[0038] S22: Based on the motion estimation results, extract pixels from the low frame rate video frames to determine the interpolation results.

[0039] In one exemplary embodiment, S3 specifically includes the following steps.

[0040] S31: Input the low frame rate video frame and the interpolation result into the color correction module, and stitch them together in the channel dimension to determine the stitched video frame.

[0041] S32: Input the spliced ​​video frames into the deep learning network, estimate the color deviation of the frame interpolation result by random perturbation of the color space, and determine the color deviation estimation result.

[0042] S33: Based on the color deviation estimation result, the interpolation result is corrected to determine the final interpolation result.

[0043] In one exemplary embodiment, S32 specifically includes the following steps.

[0044] S321: The stitched video frames are input into the deep learning network, converted to the target color space using DaVinci Resolve software, and the video frames and supervision signals after random color space conversion are determined based on the color space conversion table. The video frames after random color space conversion are obtained by color space conversion of the low frame rate video frames. The color space conversion table shows the conversion methods for different color spaces and is calculated by the algorithm.

[0045] S322: Estimate the color deviation of the interpolation result based on the video frames after random color space conversion and the monitoring signal, and determine the color deviation estimation result.

[0046] In one exemplary embodiment, to ensure the algorithm stably supports different color spaces, this application not only supervises the frame interpolation results during the second stage of training, but also re-maps the frame interpolation results using DaVinci Resolve based on the selected color space. Returning to the RGB space for training target loss calculation, S33, and then including:

[0047] S34: Using DaVinci Resolve software, perform inverse mapping based on the selected color space to convert the final frame interpolation result back to the RGB color space, and calculate the training target loss; wherein, the training target loss is:

[0048]

[0049] in, Loss due to training objective; The loss function is based on reconstruction, and it is derived from... norm ( The norm and perceptual loss (LPIPS Loss) are obtained through weighted calculation; This serves as a reference result for conversion to different color spaces; for The estimated value; T is the function mapping for converting from RGB color space to different color spaces.

[0050] The training scheme for the video frame interpolation model in S1 is divided into two stages. The first stage is the video frame interpolation model training stage, i.e., the video frame interpolation algorithm training stage, as shown in Figure 2. In order for the video frame interpolation algorithm to accurately analyze the motion between input frames and estimate intermediate frames, the algorithm does not consider the influence of different color spaces on the interpolation results during the first stage of training. Therefore, this application temporarily removes the color correction module from the video frame interpolation model and uses high frame rate video frames in the RGB color space as training data. The high frame rate video frames are extracted to obtain low frame rate input, and the extracted video frames are used as supervision signals for training.

[0051] Given low frame rate video frames as input The video frame interpolation model estimates the motion of the input low frame rate video frames and extracts pixels from the two frames accordingly to obtain the interpolation result. .

[0052] The second stage integrates a color correction module. As shown in Figure 3, the two-stage video frame interpolation training scheme, after the first stage of training, yields a stable video frame interpolation model. Before starting the second stage of training, all model parameters from the first stage are frozen, and the color correction module is integrated into the video frame interpolation model. During training, the color spaces of the input RGB data and the supervised data are randomly converted using the following operations.

[0053] Randomly select commonly used color space formats in movie video frame interpolation, such as XYZ-E, XYZ-D65, YUV, etc., and convert the input RGB video to other color spaces using DaVinci Resolve software. This conversion is denoted as... Where C is the converted image, This is a function mapping that converts an image I in the RGB color space to different color spaces.

[0054] Given RGB input and monitoring signals The video frames and monitoring signals after random color space conversion are denoted as follows: The interpolation result after random color space conversion is , for Intermediate results.

[0055] To ensure compatibility with unpredictable customized color spaces in the future, the color space conversion table is randomly shuffled, introducing a degree of randomness during color conversion so that the algorithm is not limited to the selected color space.

[0056] The color correction module aims to correct the colors of the interpolated frame results to ensure that negative effects such as color shifts due to color space limitations do not occur during final playback. To accommodate different algorithm designs and network architectures, this application places the color correction module within the video interpolation model to obtain the interpolated frame results. after.

[0057] Will and interpolation results As input to the color correction module: This application simply concatenates the three input images along the channel dimension and then feeds them into the deep learning network F(…). In this process, the deep learning network is enabled to simultaneously acquire the input and output frames of the video frame interpolation model. The deep learning network then estimates the color deviation of the interpolated results and corrects them. To ensure that the color correction module primarily corrects and optimizes colors, without excessively adjusting the motion and content of the interpolated frames, the deep learning network does more than just estimate the color correction amount. Simultaneously estimate the attenuation coefficient The interpolation results are then processed using the following formula. Perform corrections.

[0058]

[0059] in; This is the final frame interpolation result; This is the result of frame interpolation; This is the amount of color correction.

[0060] In practical applications, film production companies integrate the packaged video frame interpolation SDK into their relevant software. By opening the SDK entry point and using the film frame interpolation method provided in this application, the input film video is interpolated to obtain a high frame rate video output.

[0061] For inputs in different color spaces, such as RGB, YUV, and XYZ, this application does not require external conversion to a color space completely consistent with that used during training. Instead, it optimizes the algorithm by using a proposed color correction module to correct color shift issues caused by the mismatch between color spaces during training and actual use. This approach is more stable and efficient.

[0062] This application proposes a two-stage video frame interpolation training scheme, which can quickly and stably train the video frame interpolation algorithm and color correction module. Simultaneously, the algorithm introduces a certain degree of randomness in color space mapping during the training process, better supporting unpredictable color spaces that may be used in future film post-production, and exhibiting strong algorithm generalization capabilities.

[0063] This application provides a movie frame interpolation system, which includes the following modules.

[0064] The training module is used to train the video frame interpolation model using high frame rate videos in the RGB color space as training data, and to determine the trained video frame interpolation model.

[0065] The interpolation result determination module is used to estimate the motion of the input low frame rate video frames using the trained video interpolation model and determine the interpolation result.

[0066] The correction module is used to input the low frame rate video frames and the interpolation results into the color correction module, and correct the interpolation results through random perturbation of the color space to determine the final interpolation result. The color correction module is used to enhance the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged, so that the enhanced video interpolation algorithm can interpolate frames in different color spaces and obtain interpolation results without negative effects. As a post-processing module for interpolation, the color correction module randomly converts the input RGB data to different color spaces during training, so that the RGB data can correct the negative effects of interpolation in different color spaces. The RGB data is high frame rate video in the RGB color space, and the different color spaces include XYZ color space and YUV color space. The XYZ color space includes XYZ-E and XYZ-D65.

[0067] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments. The computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device stores data to be processed. The I / O interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with an external terminal via a network connection. When the computer program is executed by the processor, it implements the above-described methods.

[0068] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0069] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0070] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0072] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0073] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for interpolating movie frames, characterized in that, include: High frame rate videos in the RGB color space are used as training data to train the video frame interpolation model and determine the trained video frame interpolation model. The training video interpolation model is used to estimate the motion of input low frame rate video frames to determine the interpolation result. The low frame rate video frames and the interpolation result are then input into a color correction module. The interpolation result is corrected using a random perturbation method in the color space to determine the final interpolation result. Specifically, this includes: inputting the low frame rate video frames and the interpolation result into the color correction module and splicing them along the channel dimension to determine the spliced ​​video frame; inputting the spliced ​​video frame into a deep learning network and estimating the color deviation of the interpolation result using a random perturbation method in the color space to determine the color deviation estimation result. Specifically, this includes: inputting the spliced ​​video frame into the deep learning network, converting it to the target color space using DaVinci Resolve software, and determining the randomly converted color space video frame and the supervision signal based on a color space conversion table; the randomly converted color space video frame is obtained by color space perturbation of the low frame rate video frame. The process involves several steps: first, the color space is converted to a random color space; second, the color deviation of the interpolated frame result is estimated based on the video frames after random color space conversion and the supervision signal, and the color deviation estimation result is determined; third, the interpolated frame result is corrected based on the color deviation estimation result to determine the final interpolated frame result; fourth, the color correction module is used to enhance the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged, so that the enhanced video interpolation algorithm can interpolate frames in different color spaces and obtain interpolation results without negative effects; fifth, the color correction module, as a post-processing module for interpolation, randomly converts the input RGB data to different color spaces during training, so that the RGB data can correct the negative effects of interpolation in different color spaces; sixth, the RGB data is high frame rate video in RGB color space, and the different color spaces include XYZ color space and YUV color space; the XYZ color space includes XYZ-E and XYZ-D65.

2. The film frame interpolation method according to claim 1, characterized in that, The motion of the input low frame rate video frame is estimated using a trained video frame interpolation model to determine the interpolation result. Specifically, this includes: estimating the motion of the input low frame rate video frame using a trained video frame interpolation model; the low frame rate video frame includes two frames; and extracting pixels from the low frame rate video frame based on the motion estimation result to determine the interpolation result.

3. The film frame interpolation method according to claim 1, characterized in that, Based on the color deviation estimation results, the interpolation results are corrected to determine the final interpolation results. This process then includes: using DaVinci Resolve software, performing inverse mapping based on the selected color space to convert the final interpolation results back to the RGB color space, and calculating the training target loss; wherein the training target loss is: in, Loss due to training objective; The loss function is based on reconstruction. This serves as a reference result for conversion to different color spaces; for The estimated value; T is the function mapping for converting from RGB color space to different color spaces.

4. The film frame interpolation method according to claim 1, characterized in that, The final interpolation result for: in, This is the result of frame interpolation; This is the amount of color correction. The attenuation coefficient is determined by the network's variation with color correction. We can estimate it together.

5. A film frame interpolation system, characterized in that, include: The training module is used to train the video frame interpolation model using high frame rate videos in the RGB color space as training data, and to determine the trained video frame interpolation model. The interpolation result determination module is used to estimate the motion of the input low frame rate video frames using a trained video interpolation model to determine the interpolation result. The correction module is used to input the low frame rate video frames and the interpolation result into a color correction module, and correct the interpolation result using a random perturbation method in the color space to determine the final interpolation result. Specifically, this includes: inputting the low frame rate video frames and the interpolation result into the color correction module, splicing them along the channel dimension to determine the spliced ​​video frame; inputting the spliced ​​video frame into a deep learning network, estimating the color deviation of the interpolation result using a random perturbation method in the color space, and determining the color deviation estimation result. Specifically, this includes: inputting the spliced ​​video frame into the deep learning network, converting it to the target color space using DaVinci Resolve software, and determining the randomly converted color space video frame and the supervision signal based on a color space conversion table; the randomly converted color space video frame is derived from the low frame rate video frame. The interpolation results are obtained by converting video frames to different color spaces. The color deviation of the interpolated results is estimated based on the randomly converted video frames and the monitoring signal to determine the color deviation estimation result. The interpolation results are then corrected based on the color deviation estimation result to determine the final interpolation result. The color correction module enhances the video interpolation algorithm while keeping the parameters of the trained video interpolation model unchanged, enabling the enhanced algorithm to interpolate frames in different color spaces and obtain interpolation results without negative effects. As a post-processing module for interpolation, the color correction module randomly converts the input RGB data to different color spaces during training, allowing the RGB data to correct for negative effects in interpolation across different color spaces. The RGB data refers to high frame rate video in the RGB color space, and the different color spaces include XYZ and YUV color spaces. The XYZ color space includes XYZ-E and XYZ-D65.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the cinematic frame interpolation method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the movie frame interpolation method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the movie frame interpolation method according to any one of claims 1-4.

Images