System and method for multi-frame video frame interpolation

Abstract

Systems and methods for multi-frame video frame interpolation are provided. High order motion modeling, such as third order motion modeling, enables prediction of intermediate optical flow between multiple interpolated frames by relaxing constraints imposed by loss functions used in initial optical flow estimation. A temporal pyramid optical flow refinement module coarsely-to-finely refines optical flow maps used to generate intermediate frames, thereby concentrating proportionally more refinement attention on optical flow maps of high error intermediate frames. A temporal pyramid pixel refinement module coarsely-to-finely refines generated intermediate frames, thereby concentrating proportionally more refinement attention on the high error intermediate frames. A generative adversarial network (GAN) module computes loss functions used to train neural networks used in the optical flow estimation module, the temporal pyramid optical flow refinement module, and / or the temporal pyramid pixel refinement module.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Non-Provisional Patent Application No. 17 / 102,114, filed November 23, 2020, entitled "Systems and methods for multi-frame video frame interpolation," and U.S. Provisional Patent Application No. 62 / 985,524, filed March 5, 2020, entitled "System, device and method for multi-frame video frame interpolation," the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to video frame interpolation, and more particularly, to systems and methods for interpolating multiple video frames. Background Technology

[0004] Video frame interpolation

[0005] Video frame interpolation is used to synthesize frames (called intermediate frames) of digital video that occur at a time step between two existing consecutive frames of the digital video. Given two existing consecutive frames (start frame and end frame) of digital video, the video frame interpolation system synthesizes one or more intermediate frames (interpolated intermediate frames) that fill the transition between the start frame and the end frame.

[0006] Figure 1A A digital video is shown, comprising an existing start video frame 102 (start frame 102), an existing end video frame 104 (end frame 104), and an interpolated intermediate video frame 106 synthesized by a video frame interpolation system (interpolated intermediate frame 106). More generally, the video frame interpolation system synthesizes the interpolated intermediate frame 106, which includes an image representing the moment of an object between two consecutive frames in the digital video (i.e., start frame 102 and end frame 104).

[0007] Most video interpolation methods combine the pixel values ​​of the starting frame 102 and the ending frame 104 to determine the estimated motion of objects within frames 102 and 104, as shown below:

[0008] I t = f(I 0 ,I 1 ) (Equation 1)

[0009] in, I t It is the time between two frames. t Interpolation intermediate frame 106 at the location, I 0 is the start frame 102. I 1 is the end frame 104. The pixel value at coordinates (x, y) is obtained by combining the pixels at positions defined by motion vectors (u0, v0) and (u1, v1), which correspond to I from I0 and I1 to time t. t Sports:

[0010] I t (x, y) = f(I 0 (x–u 0 , y–v 0 ), I 1 (x–u 1 , y–v 1 ) ) (Equation 2)

[0011] Recent video frame interpolation methods use models to synthesize interpolated intermediate frames based on a start frame and an end frame. These models are constructed or built using machine learning algorithms based on sample video frames (hereinafter referred to as frames) of digital video. For example, the weights and biases of such neural network models are learned during training by feeding inputs with known desired outputs into a neural network model. During the training of the neural network model, frames 1 and 3 of the existing video are input into the neural network, which synthesizes and outputs the interpolated intermediate frame based on frames 1 and 3. The interpolated intermediate frame is compared with frame 2 of the existing video according to an error function (i.e., a loss function). This comparison produces an error (also called loss) that indicates how much the synthesized interpolated frame output by the neural network model deviates from the ground truth (e.g., frame 2). Backpropagation algorithms are used to adjust (e.g., update) the parameters of the neural network model. This process can be repeated using a large dataset of video frames until the model's parameters are tuned to produce an output that highly matches the desired output. That is, when fed frames 1 and 3 of the existing video, the trained neural network model synthesizes an interpolated intermediate frame that highly matches frame 2 of the existing video. At this stage, the neural network model can be called the training model.

[0012] Recent video frame interpolation methods are typically vector-based or kernel-based. Vector-based video frame interpolation methods rely on estimating bidirectional dense optical flow from a pair of consecutive frames of the video. Then, using the estimated optical flow between this pair of consecutive frames, pixel remapping is performed to synthesize interpolated intermediate frames between these frames. These interpolated intermediate frames are typically synthesized according to the following equation:

[0013] (Equation 3)

[0014] in, I 1 and I 2 represents the start frame and the end frame. I t It is an interpolated intermediate frame. x and y These are pixel coordinates. m It is a fusion mask. u and v It is the optical flow vector.

[0015] As an alternative to vector-based video frame interpolation methods, kernel-based video frame interpolation methods synthesize pixel values ​​for interpolated intermediate frames by performing convolutions on any input patch of a pair of sequentially consecutive frames of a digital video using a prediction space adaptive kernel. Specifically, for each pixel in the target interpolated intermediate frame, the kernel-based method generates a kernel, such as a rectangular 41×41 matrix. Finally, the pixel values ​​in the output interpolated intermediate frame are computed by applying the kernel to the neighborhood of the starting and ending frame pixels using a dot product.

[0016] Figure 1B The basic operations of kernel-based and vector-based methods are illustrated. The vector-based video frame interpolation method 120 applies optical flow vectors (… u, v 122, based on start frame I t 102 of ( x+u,y+v The corresponding pixel value at position 124 is used to synthesize the interpolated pixel value, thereby synthesizing the interpolated intermediate frame I. t+1 106 pixels ( x, y 126. Therefore, I t+1 ( x, y ) = f (I t ( x+u, y+ v )).

[0017] The kernel-based video frame interpolation method 130 uses kernel K( x, y ) 132. Dot product applied to the starting frame I t 102 pixels ( x, y The neighborhood P(134) x, y ) 138 to generate interpolated intermediate frames I t+1 106 pixels ( x, y 136. Therefore, I t+1 ( x, y ) = K( x, y ) ● P( x, y ).

[0018] Multi-frame interpolation

[0019] Vector-based and kernel-based frame interpolation methods have been applied to the problem of generating multiple intermediate frames, known as multi-frame interpolation.

[0020] Figure 2A An example of multi-frame interpolation is shown, starting with a frame sequence of digital video (hereinafter referred to as video sequence 100). Video sequence 100 includes a start frame (start frame 102) and an end frame (end frame 104). Multi-frame interpolation generates an intermediate frame 106 not only at the midpoint between start frame 102 and end frame 104, but also at various timestamps between start frame 102 and end frame 104. Here, seven intermediate frames have been generated, namely the first intermediate frame 108 located at one-eighth of the time distance from start frame 102 to end frame 104, and the seventh intermediate frame 110 located at seven-eighths of the time distance from start frame 102 to end frame 104. The fourth intermediate frame is intermediate frame 106. In the context of this specification, the timestamp of start frame 102 is referred to as t=0, and the timestamp of end frame 104 is referred to as t=1. Therefore, the timestamp of the first intermediate frame 108 is 0.125, the timestamp of the intermediate frame 106 is 0.5, and the timestamp of the seventh intermediate frame 110 is 0.875.

[0021] The training technique used in multi-frame interpolation methods employing models constructed via machine learning algorithms is similar to that described above for single-frame interpolation. During training, discontinuous start and end frames are selected from the video sequence in the training dataset, and these two frames are fed as input to the supervised machine learning algorithm. The actual (ground truth) intermediate frames in the video sequence from the training data between the two discontinuous input frames are used as ground truth data to calculate the loss used to adjust the model parameters. This process is repeated multiple times using a large set of start and end input frames and their corresponding ground truth intermediate frames.

[0022] Multi-frame interpolation methods applied to digital video can produce interpolated video sequences that can be used as slow-motion digital video, adaptive video streaming, video sequences reconstructed to replace missing or corrupted frames, or high-frame-rate video sequences for high-frame-rate displays. Generally, higher frame rates provide higher-quality video with smoother motion. For example, inserting a new frame between two sequential frames in a digital video frame sequence to slow down or smooth the playback by a factor of two can generate more than one new frame and insert it between two sequential frames in the frame sequence to make the playback even slower or smoother.

[0023] As described above, a single-frame interpolation method is based on estimating the optical flow between a pair of consecutive frames (e.g., two sequential frames) of a frame sequence of a digital video and linearly extending the motion to intermediate frames to obtain the positions of pixels in the intermediate frames. In multi-frame interpolation, the motion between a starting frame or an ending frame and an intermediate frame (i.e., the frame corresponding to a timestamp in the middle between the timestamp of the starting frame and the timestamp of the ending frame) is typically mapped as follows: assuming a linear transition between the starting frame and the ending frame and using a linear model to extend the motion vectors according to the relative time difference. However, the real-world motion reflected in the frames of a frame sequence of a digital video follows various complex non-linear trends, resulting in a difference between the actual motion and the reconstructed motion in the interpolated frames.

[0024] Generating multiple frames can utilize a single-frame interpolation scheme through a recursive method or a timestamp insertion method. In the recursive method, intermediate frames are inserted between two sequential frames multiple times. A typical single-frame interpolation method for generating one intermediate frame 106 (i.e., at the intermediate time step of a given pair) is used to generate a frame between two sequential frames of a digital video. Then, in the next step, for each pair of an initial frame and a generated frame, an additional intermediate frame is recursively generated.

[0025] Figure 2B An example of recursive frame interpolation for inserting seven new frames is shown, where, at the first level 112, a new frame (intermediate frame 106 at t = 0.5) is generated in the middle, at the second level 114 two intermediate frames are generated, and finally at the third level 116, an intermediate frame is inserted between each pair of frames, thereby generating 4 additional new frames, resulting in a total of seven frames.

[0026] In the recursive multi-frame interpolation method, operations can be performed in different order for different levels to achieve the desired number of generated frames. It is usually not possible to run these operations in parallel because each lower level depends on the frames generated by the higher level. When generating a new frame, it also only uses information from two adjacent frames. Another drawback of the recursive multi-frame interpolation method is that it is not flexible enough in terms of the number of frames to be generated: the number of generated frames is usually required to be 2 n–1 to the power of n, where n is the number of levels.

[0027] Other methods for generating multiple intermediate frames with arbitrary timestamps between a starting frame and an ending frame. Based on the assumption of linear movement from the starting frame to the ending frame, new intermediate frames can be generated one by one at the corresponding timestamps. These methods set a parameter to control the timestamps of the interpolated frames.

[0028] In these methods, a new intermediate frame is inserted at any time t (0 < t < 1) between the starting frame 102 and the ending frame 104. To generate multiple new intermediate frames between a given pair of frames in this method, the timestamps corresponding to each new intermediate frame are used to generate the new intermediate frames, as Figure 2C As shown. This method does not have the same limitations as recursive multi-frame interpolation methods that operate in parallel; however, frame generation optimization cannot use information from one newly generated intermediate frame to optimize another, and the operations for each new intermediate frame are independent of the operations for other intermediate frames. From a computational cost perspective, it may not be efficient, and there may be temporal inconsistencies and non-smoothness between the generated new intermediate frames.

[0029] In view of the above, we hope to improve the multi-frame video interpolation method. Summary of the Invention

[0030] This invention provides a system and method for video frame interpolation. Considering the challenges faced by current single-frame interpolation methods for generating multiple new intermediate frames, such as complexity, flexibility, and temporal inconsistency, there is a need for a method to generate all new intermediate frames in a single processing step with high temporal consistency.

[0031] Some embodiments employ higher-order motion modeling than existing multi-frame video frame interpolation methods, such as third-order motion modeling (compared to second-order or linear motion modeling used in existing methods), to more accurately predict intermediate optical flow between multiple interpolated new intermediate frames in a digital video frame sequence between the start and end frames. The frame sequence of digital video is hereinafter referred to as a video sequence. In some embodiments, higher-order motion modeling is used in conjunction with relaxation of constraints imposed by the loss function used in the initial optical flow estimation. Some embodiments use a temporal pyramid optical flow correction module to coarsely-fine correct the optical flow graph used to generate (e.g., interpolate) new intermediate frames between the start and end frames of the video sequence, proportionally focusing more correction attention on the optical flow graph of the highest-error intermediate frame (i.e., the intermediate frame with a timestamp closest to the midpoint between the start and end frame timestamps). Some embodiments use a temporal pyramid pixel correction module to coarsely-fine correct the generated new intermediate frames, proportionally focusing more correction attention on the highest-error intermediate frame. Some embodiments use modules that implement generative adversarial networks (GANs) to compute the loss used to train neural networks implemented in the optical flow estimation module, the temporal pyramid optical flow correction module, and / or the temporal pyramid pixel correction module.

[0032] Some embodiments can improve upon the limitations of existing video frame interpolation methods. Some embodiments can generate multiple new intermediate frames between the start and end frames of a video sequence in a single processing step. Some embodiments can consider temporal consistency by applying adaptive processing to the optical flow graph used to generate the intermediate frames and / or the generated intermediate frames themselves, focusing the processing on the highest-error intermediate frames. Some embodiments can apply a higher degree of motion modeling than existing multi-frame video frame interpolation methods.

[0033] Some embodiments can provide fast and parallel multi-frame video interpolation. In some embodiments, using a temporal pyramid correction module to correct the optical flow map and / or intermediate frames can take into account the error level of the intermediate frames as a function of their temporal distances to the start and end frames. Some embodiments can provide a video interpolation system that is scalable to generate (e.g., interpolate) any number of new intermediate frames between the start and end frames of a video sequence. In some embodiments, pyramid processing can be used for any temporal processing of frames in a video sequence that takes into account error propagation. Some embodiments can perform fast multi-frame video interpolation methods to generate ultra-slow-motion video.

[0034] As used herein, the term "optical flow" refers to the representation of the motion of an object (e.g., a pixel) between one video frame and another. Thus, for example, if a particular pixel is located at (x, y) coordinates (100, 100) in the first video frame and (110, 100) in the second video frame, the optical flow from the first frame to the second frame at position (100, 100) can be represented as a vector (10, 0). A two-dimensional graph showing the optical flow at each pixel position in the first frame can be called an "optical flow graph" representing the optical flow from the first frame to the second frame; in this example, the optical flow graph can be represented as a vector field with a value of (10, 0) at position (100, 100). This optical flow graph showing the optical flow from the first frame to the second frame, assuming the first frame corresponds to a time point preceding the second frame, can be called a representation of "forward optical flow." A second optical flow graph representing "backward optical flow" (i.e., the movement of an object between the second and first frames) can also be calculated based on the positions of specific pixels in the second frame and their corresponding positions in the first frame. A combination of a forward optical flow graph and a backward optical flow graph can be referred to as a "bidirectional optical flow graph," while a single forward or backward optical flow graph can be referred to as a "unidirectional optical flow graph." The embodiments described herein may use either bidirectional or unidirectional optical flow graphs, but bidirectional optical flow graphs can provide additional information that may improve the accuracy or performance of the described embodiments. It can be assumed that the term "optical flow graph" refers to either a unidirectional or bidirectional optical flow graph.

[0035] As used herein, the terms "intermediate frame" and "outer frame" refer to intermediate frames generated using interpolation between the start and end frames of an existing video sequence. An "intermediate" frame is an interpolated intermediate frame that is (temporally) relatively far from both the start and end frames, while an "outer frame" is (temporally) relatively close to either the start or end frame. Therefore, in the example of interpolating seven intermediate frames (frames 1 to 7) between the start frame at time t=0 and the end frame at time t=8, the seven intermediate frames (temporally) are located from t=1 to t=7, with frames 1 and 7 considered the "outermost" frames and frame 4 considered the "middlemost" frame. The embodiments described herein can allocate relatively more processing resources to correcting intermediate frames compared to outer frames because the temporal distance between intermediate frames and the ground truth (i.e., the start and end frames) often results in higher interpolation errors, thus requiring more correction. Attached Figure Description

[0036] Embodiments will now be described by way of example with reference to the accompanying drawings, wherein the same reference numerals may be used to indicate the same features.

[0037] Figure 1A It is a diagram showing a digital video including the start frame, interpolated intermediate frames, and end frame.

[0038] Figure 1B This is a diagram of vector-based video frame interpolation methods and kernel-based video frame interpolation methods.

[0039] Figure 2A It is a diagram showing a video sequence including a start frame and an end frame, which are used to generate seven interpolated intermediate frames.

[0040] Figure 2B This is a schematic diagram illustrating a recursive intermediate frame insertion method for multi-frame interpolation.

[0041] Figure 2C This is a schematic diagram illustrating a timestamp-based multi-frame interpolation method.

[0042] Figure 3A This is a graph showing the frame index of multiple interpolated frames relative to the peak signal-to-noise ratio measured in decibels, illustrating the performance of three known multi-frame interpolation techniques.

[0043] Figure 3B It is a graph of the time index relative to the position of the object in the frame, showing a comparison of the actual motion with several modeled motion trajectories in several multi-frame interpolation techniques.

[0044] Figure 4 This is a block diagram of an exemplary computing system for performing multi-frame interpolation, provided by the exemplary embodiments described herein.

[0045] Figure 5This is a high-level flowchart of an exemplary method for performing multi-frame interpolation, provided by the exemplary embodiments described herein.

[0046] Figure 6 This is a block diagram of a multi-frame interpolation system provided by the exemplary embodiments described herein.

[0047] Figure 7 These are performance graphs of various motion models provided by the exemplary embodiments described herein.

[0048] Figure 8 This is a graph of the x and y coordinates of an object across multiple frames, based on the exemplary embodiments described herein, illustrating a comparison of actual motion with linear motion models and third-order motion models.

[0049] Figure 9 It is a graph of the object's x and y coordinates across multiple frames, showing a comparison between the actual motion and linear and second-order motion models.

[0050] Figure 10 This is a block diagram of the temporal pyramid optical flow correction module of the multi-frame interpolation system provided in the exemplary embodiments described herein.

[0051] Figure 11 yes Figure 9 A block diagram of the subnetwork of the time pyramid optical flow correction module.

[0052] Figure 12 yes Figure 10 A simplified block diagram of the time pyramid optical flow correction module.

[0053] Figure 13 This is a block diagram of the temporal pyramid pixel correction module of the multi-frame interpolation system provided in the exemplary embodiments described herein.

[0054] Figure 14 This is a block diagram of a generative adversarial network used in training a multi-frame video frame interpolation system, provided by an exemplary embodiment described herein.

[0055] Figure 15 This is another block diagram of the temporal pyramid optical flow correction module of the multi-frame video frame interpolation system provided in the exemplary embodiments described herein. Detailed Implementation

[0056] As used herein, "module" can refer to a component or part of a software system, including machine-readable instructions executable by a processing unit of a computing system. "Module" can also refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable by that hardware processing circuitry. A processing unit can be a central processing unit (CPU), tensor processing unit (TPU), or neural processing unit (NPU) with a hardware accelerator. Hardware processing circuitry can include any or some combinations of microprocessors, the core of a multi-core microprocessor, a microcontroller, an application-specific programmable integrated circuit (ASIC), a field-programmable gate array (FPGA), a central processing unit (CPU), a tensor processing unit, a neural processing unit, or other hardware processing circuitry.

[0057] This invention is carried out with reference to the accompanying drawings, which illustrate embodiments. However, many different embodiments may be used, and therefore the description should not be construed as limiting oneself to the embodiments set forth herein. Rather, these embodiments are provided to make the invention thorough and complete. Where possible, the same reference numerals are used in the drawings and the following description to refer to the same elements, and prime number notation is used in alternative embodiments to indicate the same elements, operations, or steps. The separate blocks or separations of functional elements of the illustrated systems and devices do not necessarily require physical separation of these functions, as communication between these elements can occur without any such physical separation via message passing, function calls, shared memory spaces, etc. Therefore, although functions are shown separately herein for ease of explanation, these functions do not need to be implemented in physically or logically separated platforms. Different devices may have different designs such that while some devices implement some functions in fixed-function hardware, others may implement those functions in a programmable processor with code available from a machine-readable medium. Finally, elements mentioned in the singular may be plural, and vice versa, unless the context explicitly or inherently indicates otherwise.

[0058] Exemplary embodiments of software systems, computing systems, and methods for video frame interpolation, particularly multi-frame video interpolation, will now be described. Some exemplary embodiments of the software systems and computing systems use trained models to perform all or part of the video frame interpolation method. Some exemplary embodiments of the systems and devices include trained neural networks that perform all or part of the video frame interpolation method.

[0059] Existing multi-frame interpolation techniques typically treat all frames in a video sequence with equal importance; they generally do not take advantage of the expected level of difficulty in generating multiple intermediate frames between the start and end frames of a video's frame sequence (hereinafter referred to as the video sequence) at different timestamps. In fact, as motion extends, it is easier to generate intermediate frames that are closer to one of the two initial frames of the video sequence (i.e., the start and end frames) than to generate intermediate frames with a larger temporal distance. Therefore, the machine learning models implemented in existing multi-frame interpolation systems are not optimized in terms of model size (i.e., the number of parameters the model learns) or execution time, making them unsuitable (or unoptimized for) user-facing software applications. To address this, some embodiments described herein utilize temporal pyramid processing operations to efficiently integrate multi-frame generation into a single neural network. The described embodiments can adaptively process low-error intermediate interpolated frames (e.g., frames close to the start or end frames) with shallow processing steps to guide the generation of high-error intermediate interpolated frames (e.g., frames far from the start and end frames) processed by deeper processing steps. By jointly optimizing all intermediate frames, higher quality interpolated intermediate frames can be synthesized, which have higher temporal consistency across the intermediate interpolated frame sequence.

[0060] Furthermore, most existing multi-frame video interpolation methods synthesize intermediate frames by simply assuming a linear transition of motion between the start and end frames. However, the real-world motion reflected in video frames follows various complex nonlinear trends. To overcome this limitation, an extension of the second-order motion prediction model can be used. However, assuming constant acceleration between input frames, the second-order motion prediction model may still be insufficient to model the real-world motion of objects in some cases, especially for non-rigid objects. In the real world, the forces applied to a moving object are not necessarily constant, causing variations in the acceleration of the object or parts thereof. To address this issue, some embodiments can leverage the advantages of multi-frame input to implement a high-level, higher-order motion prediction model (e.g., third-order instead of second-order) that accounts for changes in acceleration. Additionally, some embodiments can improve the quality of motion prediction and the final interpolated intermediate frames by using a relaxation loss function during the training of the optical flow estimation neural network in the optical flow estimation module. This provides the flexibility needed to map pixels to the neighborhood of their ground truth location at the reference frame, while achieving better motion prediction for the intermediate frames. Some implementations can also optimize the size (i.e., the number of learned parameters of the trained neural network) and runtime of optical flow estimation neural networks, making them suitable for real-world applications, especially on mobile devices.

[0061] It can be observed that there is a temporal distribution of errors in multi-frame video interpolation.

[0062] like Figure 3A As shown, the seven interpolated intermediate frames, spaced apart along the horizontal X-axis 302 by frame indices 1 to 7, have different peak signal-to-noise ratio (PSNR) levels as shown on the vertical Y-axis 304. Curves for each of the three known multi-frame video frame interpolation methods, DAIN 306, SepConv 308, and CyclicGen 310, are plotted, showing a consistent pattern where the most intermediate interpolated frame (i.e., the frame closest to the midpoint between the start frame 102 and the end frame 104) has the lowest PSNR and therefore the highest error level. This is expected because newer interpolated intermediate frames with timestamps closer to the start or end frames are expected to have lower error levels, while intermediate frame 104 (here, the frame with frame index 4) is temporally furthest from the ground truth frames (i.e., the start and end frames).

[0063] It can also be observed that the motion between frames may not be at a constant speed.

[0064] exist Figure 3BIn the diagram, solid curve 320 represents the actual path of an object (e.g., a pixel in a frame) in non-linear motion, and point 322 (at timestamps 330 at -4, -2, 2, and 4) represents the observations of the object's position on that path given by the four input frames. Motion modeling for video frame interpolation aims to find the object's position for each frame to be interpolated within a time interval between -2 and 2 (i.e., the predicted target interval, corresponding to the time period to be filled by the new interpolated intermediate frame). The dashed line 324 represents the direct path between observations. The seven solid points 326 in the target interval are the predicted points when modeling motion using point 322. Solid points 326 are located away from the actual path (i.e., solid curve 320).

[0065] Exemplary embodiments of the software systems, computing systems, and methods described herein for video interpolation, particularly multi-frame video interpolation, can address one or more of these observations to improve the performance of multi-frame video interpolation. Some embodiments use higher-order motion prediction models than existing multi-frame video interpolation methods to model the motion of intra-frame objects, such as third-order motion modeling (as opposed to, for example, linear or second-order motion modeling), to more accurately predict intermediate optical flow between multiple interpolated intermediate frames. In some embodiments, higher-order motion modeling is used in conjunction with relaxing the constraints imposed by the loss function used in training the optical flow estimation neural network implemented by the optical flow estimation module. Some embodiments use a temporal pyramid optical flow correction module to perform coarse-to-fine correction on the optical flow map used to generate intermediate frames between the start and end frames, by proportionally focusing less correction attention on the optical flow map of the low-error outermost frame (i.e., coarse correction) and proportionally focusing more correction attention on the optical flow map of the high-error intermediate frame. Some embodiments use a temporal pyramid pixel correction module to perform coarse-to-fine correction on the generated intermediate frames, proportionally focusing more correction attention on the high-error intermediate frames. Some embodiments use generative adversarial networks (GANs) to compute the loss function used during the training of the neural networks implemented in the optical flow estimation module, the temporal pyramid optical flow correction module, and / or the temporal pyramid pixel correction module. These modules will be described in detail in their respective sections.

[0066] Computing System

[0067] Figure 4 Selected components of a computing system 400 for performing video frame interpolation, particularly multi-frame video interpolation, according to exemplary embodiments of the present invention are shown. In various embodiments, the computing system 400 may be a physical computer (i.e., a physical computer such as a desktop computer, laptop computer, server, etc.) or a virtual computer (i.e., a virtual machine) provided, for example, by a cloud service provider. Figure 4As shown, the computing system 400 includes a processor 402 coupled to a memory 426 via a communication bus or communication link 404, which provides a communication path between the memory 426 and the processor 402. In some embodiments, the memory 426 may be random access memory (RAM), read-only memory (ROM), persistent (non-volatile) memory, such as erasable programmable read-only memory (EPROM) (flash memory). The processor 402 may include one or more processing units, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), and other processing units. The processor 402 may also include one or more hardware accelerators.

[0068] The memory 426 of the computing system 400 stores instructions including video frame interpolation instructions 462, which can be executed by the processor 402 to implement a multi-frame video frame interpolation software system 700 including its various functional modules, as referred to below in the multi-frame interpolation section. Figure 6 Further described. In various embodiments, the video frame interpolation instruction 462 includes one or more of the following instructions: flow estimation instruction 464, motion modeling instruction 466, optical flow correction instruction 468, and / or post-correction instruction 470. When executed by the processor 402, the flow estimation instruction 464 causes the computing system 400 to operate the flow estimation module 704, as further described in the flow estimation section below. When executed by the processor 402, the motion modeling instruction 466 causes the computing system 400 to operate a motion modeling module, such as a higher-order motion modeling module 708, as referred to in the higher-order motion modeling section below. Figures 7 to 9 Further described. When the optical flow correction instruction 468 is executed by the processor 402, it causes the computing system 400 to operate the optical flow correction module 712, as referred to in the optical flow correction section below. Figures 10 to 12 Further described. When the post-correction instruction 470 is executed by the processor 402, it causes the computing system 400 to operate the pixel correction module 716, as referred to in the pixel correction section below. Figure 13 and Figure 14 Further description.

[0069] Memory 426 also stores various types of data 480. Data 480 may include video data 482 (i.e., data representing digital video), which includes data representing the start frame of the video (hereinafter referred to as start frame 486) and data representing the end frame of the video (hereinafter referred to as end frame 488). Start frame 486 and end frame 488 are a pair of consecutive frames of the video. Instructions including video frame interpretation instructions 462, flow estimation instructions 464, motion modeling instructions 466, optical flow correction instructions 468, and post-correction instructions 470 may be temporarily loaded into volatile memory (e.g., RAM of memory 426) for storing runtime data variables and other types of data and / or information. Data received by the multi-frame video frame interpolation software system 700 implemented by video frame interpolation instructions 462 may also be stored in the RAM of memory 426. While specific functions of various types of memory have been described, this is only an example, and different functional allocations to memory types may also be used.

[0070] The computing system 400 may be a single device, such as a collection of circuits housed within a single enclosure. In other embodiments, the computing system 400 may be distributed across two or more devices or enclosures, potentially spatially separated from each other. The communication bus 404 may include one or more communication links or networks.

[0071] Machine Learning

[0072] Machine learning (ML) is an artificial intelligence technique in which algorithms are used to construct or build a "model" for a specific task from sample data that can be applied to new input data to perform a specific task (i.e., to make predictions or decisions based on new input data) without explicitly programming the specific task.

[0073] As used herein, "model" refers to a machine learning model. A machine learning model is an executable computational construct, such as processor-executable software instructions. During model training, the model's parameters are learned using sample data (e.g., data from the training dataset). After training, the trained model can be deployed and operated in inference mode (e.g., applied to new input data) to perform a specific task (i.e., make predictions or decisions based on new input data). The machine learning models described herein can be simulated by convolutional neural networks that have been trained to perform tasks (particularly video frame interpolation). However, it should be understood that various embodiments of the systems, devices, and methods described herein are equally applicable to other tasks, other neural network architectures (e.g., fully connected or recurrent neural networks), and other machine learning techniques, including other deep learning techniques, with appropriate modifications to certain operations. Furthermore, some embodiments of the software systems, computational systems, and methods described herein can be applied to domains outside the machine learning context.

[0074] Multi-frame interpolation - Overview

[0075] Exemplary embodiments of software systems, computing systems, and methods for video frame interpolation, particularly multi-frame video frame interpolation, will now be described. Some embodiments of the software systems and computing systems disclosed herein use one or more trained models to perform all or part of the video frame interpolation method, particularly the multi-frame interpolation method.

[0076] Figure 5 A high-level flowchart of a first method 600 for multi-frame interpolation is shown, which is implemented by video frame interpolation instructions 462. Figure 6 The multi-frame video frame interpolation software system 700 is executed. The multi-frame video frame interpolation software system 700, implemented by video interpretation instruction 462, and various submodules implemented by instructions 464, 466, 468, and 470 can execute various steps (i.e., operations) of method 600, as referred to later in this section. Figure 6 and Figure 15 More detailed description. In step 602, frames of the video sequence are received. These frames include the start frame at t=0. I The end frame at 0102 and t=1 I 1104. Optionally, two additional frames can be received: the frame preceding t=–1 (i.e., before the start frame 102) and the frame following t=2 (i.e., after the end frame 104). (As mentioned above regarding...) Figure 3B As described, the previous and next frames can be used to perform high-order motion modeling on the estimated motion of objects on two or four frames of a video sequence.

[0077] In step 604, the optical flow estimation module 704 estimates the optical flow between the start frame 102 and the end frame 104 and generates a pair of optical flow maps (i.e., bidirectional optical flow maps, including a forward unidirectional optical flow map representing motion from t=0 to t=1 and a backward unidirectional optical flow map representing motion from t=1 to t=0). The optical flow estimation module 704 includes a convolutional neural network (CNN), as described in detail in the flow estimation section below. In the described embodiment, optionally, the loss function used to train the CNN included in the optical flow estimation module 704 can relax the optical flow estimation constraints during training so that the output of the optical flow estimation module 704 can provide the flexibility required for subsequent higher-order motion modeling performed in step 606.

[0078] In step 606, the motion modeling module 708 (e.g., a higher-order motion modeling module) is used to estimate... I 0102 and I The motion of objects (e.g., pixels) between 1104 is calculated, and an estimated intermediate optical flow 710 is generated between intermediate frames based on the estimated motion, as described in detail in the Motion Modeling section below. In some embodiments, third-order motion modeling may be used instead of conventional linear or second-order motion modeling.

[0079] In the example described in this article, the number of intermediate frames is 7, specified as I t1 arrive I t7 It should be understood that any number of intermediate frames can be generated using the multi-frame video interpolation method described in this paper.

[0080] In step 608, pyramidal temporal optical flow correction is used to correct the intermediate optical flow map 710, as described in detail in the optical flow correction section below. Pyramidal temporal optical flow correction focuses the correction attention on the intermediate optical flow map 710 of the middle frame, which has the highest expected error level.

[0081] In step 610, a new intermediate frame is generated based on the corrected intermediate optical flow map generated in step 608.

[0082] In step 612, pyramid temporal pixel correction is used to correct the generated intermediate frame. This step may be referred to herein as “pixel correction” or “post-correction”, which means that without this post-correction step 612, additional pixel-level corrections are performed on the complete intermediate frame that can be used as the output of method 600.

[0083] It should be understood that some embodiments may omit some of these steps or use alternative implementations of various steps. In some embodiments, the relaxation of optical flow estimation constraints at step 604 and the use of higher-order motion modeling at step 606 may be omitted and replaced with conventional optical flow estimation and linear or second-order motion modeling. In some embodiments, optical flow correction step 608 may be omitted or replaced with a non-pyramid optical flow correction step. In some embodiments, pixel correction step 612 may be omitted or replaced with a non-pyramid pixel correction step. Each of these steps contributes to obtaining the final result obtained by the multi-frame video frame interpolation method 600 and may be used without one or more other steps.

[0084] Figure 6 A high-level block diagram of a multi-frame video frame interpolation software system 700 performing method 600 is shown. Input frames 702 (shown here as four frames with time indices of –1, 0, 1, and 2) are received by a convolutional neural network (CNN) of optical flow estimation module 704. Input frame 702 at time index 0 is the start frame 102; input frame 702 at time index 1 is the end frame 104; the two frames at time indices –1 and 2 are additional frames used by motion modeling module 708 to perform higher-order motion modeling, as described in the higher-order motion modeling section below. Optical flow estimation module 704 performs step 604 to generate an initial optical flow map 706 between t=0 and t=1. Higher-order motion modeling module 708 performs step 606 to generate pixel motion based on the initial optical flow map 706 using a third-order motion model. Higher-order motion modeling module 708 generates multiple intermediate optical flow maps 710. Multiple intermediate optical flow maps 710 are shown here as seven pairs of optical flow maps corresponding to the optical flow round trip to each intermediate frame and its neighborhood: that is, the intermediate optical flow map 710 includes an estimate of the first intermediate frame I. t1 Its neighborhood (starting frame I0102 and second intermediate frame I) t2 The first pair of intermediate optical flow maps between the first and second intermediate frames, and so on, are calculated until the seventh intermediate frame I is estimated. t1 Its neighborhood (sixth intermediate frame I) t6 The seventh pair of intermediate optical flow diagrams between the optical flow of the end frame (I0104) and the end frame (I0104).

[0085] The intermediate optical flow map 710 is corrected by the optical flow correction module 712 (shown here as a pyramid temporal optical flow correction module), which performs optical flow correction step 608, as described in detail in the optical flow correction section below. The corrected optical flow map (not shown) generated by the optical flow correction module 712 is used by the frame synthesis module (not shown) to perform frame synthesis step 610 to generate intermediate frames 714 by applying the corrected intermediate optical flow map to the start frame 102 and end frame 104 of the input frame 702. After the intermediate frames 714 have been generated, these intermediate frames are further corrected by the pixel correction module 716 (shown here as a temporal pyramid pixel correction module) to perform pixel correction step 612 using the pyramid configuration of the pixel correction sub-network 718. In the embodiment shown herein, the pixel correction sub-network 718 is a convolutional neural network. The output of the pixel correction module 716 is a set of corrected intermediate frames 720.

[0086] Figure 15 Another flowchart illustrates the steps (i.e., operations) performed by the multi-frame video frame interpolation software system 700, further details the steps (i.e., operations) performed by the optical flow estimation module 704, and includes a frame synthesis step 1508 following the steps (i.e., operations) performed by the optical flow correction module 712. These details are described in the flow estimation and optical flow correction sections below.

[0087] Flow estimation

[0088] Existing flow estimation techniques may not efficiently utilize adjacent frames (e.g., the previous frame at t=–1 and the subsequent frame at t=2), and some techniques are limited to unidirectional optical flow estimation. In some embodiments, the optical flow estimation module 704 is a customized two-stage optical flow estimation module for performing two stages of optical flow estimation using adjacent frames (e.g., the previous frame at t=–1 and the subsequent frame at t=2) to target frame pairs ( I 0 and I 1) Provides better optical flow estimation, such as Figure 15 As shown.

[0089] The optical flow estimation module 704 includes two stages 1502 and 1506 of a similar three-scale neural network for performing optical flow estimation. Each stage 1502 and 1506 of the neural network shares two coarser levels of weights. Specifically, the first neural network for performing the first stage 1502 includes three incrementing correction levels, and the second neural network for performing the second stage 1506 includes three incrementing correction levels. The first level of the first neural network shares the weights of the first level of the second neural network, and the second level of the first neural network shares the weights of the second level of the second neural network. The first neural network for performing the first stage 1502 receives two consecutive (e.g., sequential) frames and calculates the bidirectional optical flow between the two consecutive (e.g., sequential) frames. Using the network performing the first stage, the corresponding frames (i.e., f) are used... 0→-1 Using the previous frame at t=–1 and the starting frame at t=0, and The end frame at t=1 and the subsequent frame at t=2 are used as input to estimate f. 0→-1 and ,like Figure 15 The number 1504 is shown in the diagram. Then, these two optical flows 1504 are used as... and The initial estimate, and with I 0 (start frame) and I 1 (End Frame) combination, and passed to the optimal level of the second neural network (i.e., the third level), which is used to execute the second stage 1506 to obtain and The final estimate. In parallel, 1503 and f 1→-1 The estimate of 1505 is generated by the neural network of the first stage 1502 and used by the higher-order motion modeling module 708.

[0090] Existing optical flow estimation techniques typically attempt to map pixels from their exact locations in the first frame to their locations in the second. However, precise optical flow is not tailored for many computer vision tasks, including video frame interpolation. Specifically, existing optical flow estimation techniques result in suboptimal performance in occluded regions between the first and second frames. In other words, the optimal optical flow approach as a step toward completing a further task (such as video frame interpolation) does not necessarily yield the optimal approach for that task. Similarly, strong constraints on optical flow estimation between input frames can degrade motion prediction and interpolation performance in intermediate frames, especially for complex nonlinear motion. In contrast, allowing some flexibility in estimating optical flow between input frames can provide flexibility in predicting object motion that more closely approximates the nonlinear motion patterns that might occur between the two initial frames. This flexibility can offer certain advantages, as illustrated in the following examples.

[0091] exist Figure 8 The diagram shows the results of optical flow estimation performed using a third-order motion model based on an example practical implementation. The dashed curve 901 simulates the motion path of a pixel over a time span of four frames, and the x-axis 920 and y-axis 922 are the pixel coordinates within the frame. The pixel position is given as X in the four consecutive frames. –1 906, X0910, X1914, and X2918. The goal is to find seven locations of pixels between X0910 and X1914, indicated by patterned star 902 using X0910 as a reference point in the figure. A straight line 905 connecting these locations represents the optical flow between X0910 and the other three points. A third-order model (as described in Equation 11 in the Higher-Order Motion Modeling section below) is applied to generate the predicted motion path, as shown by solid star 904.

[0092] Figure 8 The results of optical flow estimation performed using the third-order motion model shown can be compared with... Figure 9 Comparison, Figure 9 The results of optical flow estimation performed using a second-order motion model are shown. Pixel positions are given as X in three consecutive frames. –1 906, X0910, and X1914. Optical flow estimation is performed using a second-order motion model to produce the predicted motion path shown by solid star 904.

[0093] exist Figure 8 and Figure 9 As can be seen, the predicted position 904 is far from the ground truth (patterned star 902). It can be observed that this is not achieved by considering the ground truth position in the next frame ( X (1914) To estimate pixel positions in intermediate interpolated frames to fit the motion model, giving the motion model the flexibility to move to adjacent positions such as X1' and X1' can significantly improve motion prediction at intermediate positions. (Similarly, the relaxed flow estimation constraints described below can support the use of adjacent positions X1' and X1'.) –1 '908 and X2' 916 instead of X –1 906 and X2918. ) Figure 8 and Figure 9 In this process, by moving the entity star 904 to the corresponding contour star 903, the mean squared error (MSE) of the predicted position at the middle position will be significantly reduced. In some embodiments of the optical flow estimation module 704, by relaxing the loss function of the flow estimation model used by the flow estimation module 704, certain errors can be introduced into the optical flow estimation process, thus providing this flexibility.

[0094] To apply the concept of relaxing the loss function of the model used for optical flow estimation in the flow estimation module 704, some embodiments can use unsupervised learning, similar to that of Jiang, H., Sun, D., Jampani, V., Yang, MH, Learned-Miller, E., and Kautz, J. Super Slomo: High-quality estimation of multiple intermediate frames for video interpolation ( Super slomo: High quality estimation of multiple intermediate frames for video interpolation The unsupervised learning method used is described in the IEEE Conference on Computer Vision and Pattern Recognition, Proceedings, pp. 9000-9008 (2018) (the full text of which is incorporated herein by reference). However, some embodiments can be trained using a relaxed warp loss function to optimize optical flow. ,as follows:

[0095] ,for (Equation 4)

[0096] in, Indicates being Warp to the end frame used as the ground truth reference point I 1 of I 0, d The range of the neighborhood is determined. During training, the same loss function is applied to all optical flows estimated by the neural network included in the flow estimation module 704.

[0097] In some embodiments, the flow estimation module 704 includes a neural network, wherein the input frame pairs I 0 (starting frame 102) and I 1 (End frame 104) Processing is performed from low resolution to high resolution, and the optical flow estimate obtained at each low resolution is used by the next high resolution estimation neural network. Figure 15 A more detailed view of the flow estimation module 704 is shown. To better estimate the target pair ( I 0102 and I The optical flow in 1104) is estimated in the first stage 1502 of optical flow estimation performed by the flow estimation module 704. I –1 and I Between 0 and I 1 and I The preceding and subsequent optical flows (OF) between 2 f 0→-1 and 1504. Then, these previous and subsequent optical flows 1504 are used by the optical flow estimation module 704 to guide the second stage 1506 of optical flow estimation. I 0 and IOF estimation between 1 and 2 is performed, and an initial optical flow map 706 is generated, shown as a forward optical flow map and a backward optical flow map representing bidirectional optical flow.

[0098] To optimize the optical flow estimation performed by the optical flow estimation module 704, typically each neural network estimates the optical flow such that each pixel in the starting frame is mapped to an exact corresponding position in the ending frame. However, empirical results show that this constraint may not produce the best estimate of pixel positions in intermediate frames, especially in the case of nonlinear motion. Figure 3B The exemplary nonlinear motion pattern shown can be used to illustrate the impact of flexible OF estimation (i.e., allowing a certain amount of error in OF estimation) on motion prediction accuracy.

[0099] It can be observed that during the training of the neural network included in the optical flow estimation module 704, a certain degree of error is allowed, enabling the estimated OF to deviate from the path defined by the solid points 326 and move towards the path defined by the predicted patterned points 328 in the target interval (t=–2 to t=2), which better estimate the actual curve 320 than the solid points 326. Therefore, to generate more accurate intermediate motion predictions, the constraints on optical flow estimation are relaxed, and some error in optical flow estimation is allowed during the training of the neural network included in the optical flow estimation module 704. Some embodiments using the following loss function can be used to train the neural network of the optical flow estimation module 704:

[0100] (Equation 5)

[0101] in, I 0 warped (x, y) = I 0(x – △u, y – △v), and (△u, △v) is the calculated optical flow.

[0102] This relaxed loss function can be compared with the traditional optical flow estimation loss function, which calculates the end-to-end error by comparing the predicted optical flow vector (Δu, Δv) at each pixel location with the ground truth optical flow vector, and thus calculating the Euclidean distance between two vectors. Therefore, an exemplary traditional loss function is:

[0103]

[0104] in, I 0 warped (x, y) = I 0(x – △u, y – △v), I 1 is the ground truth value, and (△u, △v) is the calculated optical flow.

[0105] In some embodiments, the update of parameters for optical flow estimation of the neural network included in the optical flow module 704 uses only optical flow calculations in a small neighborhood of four frames, which can make training the model more efficient.

[0106] High-order motion modeling

[0107] In addition to the relaxed OF estimation mentioned above, the method used in motion modeling has a significant impact on the accuracy of the prediction. Figure 7 The results of several different motion modeling methods are shown.

[0108] exist Figure 7 In the diagram, three different models used to perform motion modeling are plotted with their object position predictions 820 for three motion scenarios: constant velocity 802, constant acceleration 804, and variable acceleration 806. Object positions are given at four timestamps (t=0, t=1, t=2, and t=3), indicated by circles with vertical dashed crosshairs, and the model is used to predict the object's position at t=1.5.

[0109] Linear motion scene 802 corresponds to Figure 3B Line connection point 322: Linear model 812, second-order motion model 814, and third-order model 816 all achieve the same prediction aligned with the ground truth 810. However, higher-order models 814 and 816 produce better fits to real-world situations that typically involve many nonlinear motions: in the constant acceleration scenario 804, the object position 820 predicted by linear model 812 differs from that of higher-order models 814 and 816; in the variable acceleration scenario 806, the object position 820 predicted by linear model 812 differs from that of second-order model 814, while second-order model 814 predicts an object position 820 that is further off-target than that of third-order model 816.

[0110] Therefore, using a third-order motion model for motion modeling can be a more powerful approach to capturing motion in real-world scenarios where objects face variable surfaces and accelerations. In some embodiments, the higher-order motion modeling module 708 can use a third-order motion model based on the initial optical flow graph 706, described by the following equation:

[0111] (Equation 6)

[0112] in, t It is the time to generate intermediate frames. It is input frame 702 (end frame) at t=1. I 1104) and input frame 702 at t=2 I Optical flow between frames 1104 and subsequent frames. This equation can be applied to frame sequence 702 to generate intermediate optical flow map 710.

[0113] A third-order motion model can model the motion of objects with varying acceleration. Consider starting from time 0 to the intermediate timestamp. t i The movement is This motion is modeled using a third-order model as follows:

[0114] (Equation 7)

[0115] in, v 0 、a 0 and Δa 0 are in I The estimated velocity, acceleration, and rate of change of acceleration at point 0. The acceleration term can be calculated as:

[0116] , , (Equation 8)

[0117] In the equation above, are respectively frames. I 0 and I In frame 1, a0 and a1 are calculated for each pixel. However, for corresponding pixels between these two frames, acceleration changes should be considered. Therefore, a i It was reformulated to fit the frame. I Pixel position calculation in 0 Δ a 0, as shown below:

[0118] (Equation 9)

[0119] Because when acceleration is variable v Since 0 is not constant, the above third-order motion equation can be applied to... t =1, solve using only the information calculated above. v 0:

[0120] (Equation 10)

[0121] Finally, any t [0,1] It can be represented solely based on the optical flow between two frames:

[0122] (Equation 11)

[0123] It can be calculated in the same way.

[0124] Optical flow correction

[0125] To leverage the similarity between multiple frames to assist interpolation, and also taking into account the pattern of error escalation in the middle frame, some embodiments may use adaptive joint processing as part of the optical flow correction module 712 and / or pixel correction module 716. One or both correction modules 712, 716 may use a pyramid model to successively correct the most error-prone interpolation frames.

[0126] In the optical flow correction module 712, the bidirectional optical flow map predicted above (Equation 11) is used. and The optical flow graph is calculated based on input frames 702. Initial optical flow prediction may inherit errors from optical flow estimation and third-order motion modeling, especially at motion boundaries. To improve the optical flow graph... and The optical flow correction module 712 considers the relationship between intermediate frames and processes all intermediate frames in a forward pass.

[0127] To this end, the optical flow correction module 712 includes a temporal pyramid optical flow correction module, which is used to enforce strong binding between intermediate frames, such as... Figure 10 As shown.

[0128] Figure 10 An example is shown that uses a single convolutional neural network to perform multi-frame interpolation progressively from easy to difficult. The single CNN receives a cascade of seven pairs of initial predicted optical flows (bidirectional intermediate optical flow graph 710) and adaptively modifies the optical flow graph based on the expected quality of the interpolated intermediate frames. The two closest initial frames ( I 0102 and I The intermediate frame of 1104 (here) I t1 and I t7 It is more likely to obtain higher quality interpolation results, so they are processed only by the first level 1152 in the pyramid, while other frames are processed by more levels 1154, 1156 and / or 1158 based on these frames to the initial frame. I 0102 and I Distance processing of 1104. Intermediate frame 106 (in this case) I t4 The optical flow is processed by all levels 1152, 1154, 1156, and 1158 in this architecture, as it is expected to achieve the lowest quality interpolation. To fully utilize the corrected optical flow map at each level 1152, 1154, 1156, and 1158 of the optical flow correction module 712, the frame is warped by the optical flow obtained at each level. and This warping and feedforward is then fed to the next level. This warping and feedforward helps achieve better results at the next level because... I 0 andI Compared to frame 1, the warped frame is closer to its position in the target frame of that layer in the time domain. Therefore, I 0 and I The motion between 1 and 2 consists of piecewise linear motion, with each motion measured within a short time interval.

[0129] In the proposed single CNN, in addition to optical flow, a fusion mask is also generated at each level. b ti Therefore, intermediate frames can be generated (e.g., synthesized) as follows:

[0130] (Equation 12)

[0131] in, and yes t i Corrected bidirectional optical flow map at the location. This represents element-wise multiplication. g (•,•) is a bilinear warping function, as in Liu, Z., Yeh, RA, Tang, X., Liu, Y., and Agarwala, A. View using depth voxel flow Video frame synthesis using deep voxel flow The information is presented in the proceedings of the IEEE International Conference on Computer Vision, pp. 4463-4471 (2017) (the full text of which is incorporated herein by reference).

[0132] During operation, the intermediate optical flow map 710 is transmitted to the optical flow correction module 712. For example... Figure 10 As shown, the optical flow correction module 712 has a pyramid-structured neural network that adaptively corrects the intermediate optical flow graph 710 based on the expected quality of the intermediate frame as a function of the time distance from the intermediate frame to the start frame 102 and the end frame 104. By using an optical flow correction module with pyramid flow correction and an optical flow estimation module with complex motion modeling, the estimated flow of the intermediate frames is corrected in the pyramid processing operation, while taking into account higher-order (e.g., third-order) motion models to better handle nonlinear motion patterns. The pyramid structure of the optical flow correction module 712 provides strong connections between interpolated intermediate frames to enforce temporal consistency.

[0133] As mentioned above Figure 3A The frame that is closest to the start frame 102 and the end frame 104 as described. I t1 and I t7 It is more likely to have higher quality. Therefore, I t1 and I t7Processing occurs in the first level 1152 of the pyramid, while other frames are processed by further processing steps within the pyramid based on their distance from the starting frame 102 and the ending frame 104. In this method, seven generating frames ( I t4 The intermediate frame is processed by all levels of the pyramid 1152, 1154, 1156, and 1158, as it is expected to have the lowest quality level in the interpolation. Furthermore, as pixels move closer to their positions in the intermediate frame over time steps, the warped frame generated by the warping operation 1104 is fed to the next level for fusion.

[0134] from Figure 10 Starting from the left side of the optical flow correction module 712 shown, the middle optical flow map 710 undergoes a cascading operation 1102 before being fed into the first level 1152 of the pyramid structure, which includes a first sub-network 1120 of the optical flow correction neural network, shown here as a 64-channel sub-network of a convolutional neural network. I t1 1132 and I t7 The optical flow diagram of 1144 has leakage. relu Convolutional blocks 1106 of the function are generated. These optical flow maps 1132 and 1144 are each warped by the warped function 1104 to generate warped first intermediate frames. and the warped seventh intermediate frame These warped frames are cascaded to the feature map output of the first sub-network 1120 via cascading operation 1102.

[0135] This process is repeated at the second level 1154, which includes a 48-channel second subnetwork 1122, and provides... I t2 1134 and I t6 1142 generates an optical flow map, and similarly feeds back the warped second and sixth intermediate frames to the feature map output of the second sub-network 1122.

[0136] This process is repeated at level 3 1156, which includes a 32-channel third subnetwork 1124, and provides... I t3 1136 and I t5 1140 generates an optical flow map, and similarly feeds back the warped second and sixth intermediate frames to the feature map output of the third sub-network 1124.

[0137] Level 4 1158 is the 24-channel fourth sub-network 1126, which is the intermediate frame. I t4 1138 generates an optical flow map as its output.

[0138] Figure 11 The subnet structure for each subnet 1120, 1122, 1124, and 1126 is shown. Subnet 1120 is shown as having leakage. relu The function consists of two convolutional blocks 1106, a single convolutional block 1108, addition operations 1110, a second repetition of 1106, 1108, and 1110, and has leakage. relu The final convolutional block 1106 of the function is a linear sequence. The first side channel will come from a leaky... relu The output of the first convolution block 1106 of the function is passed to the first addition operation 1110, and the second side channel passes the output from the first addition operation 1110 to the second addition operation 1110.

[0139] Figure 12 A further simplified illustration of the operation of the time pyramid optical flow correction module 712 is shown.

[0140] like Figure 15 As shown, frame synthesis step (i.e., operation) 1508 generates (e.g., synthesizes) a set of intermediate frames 714 based on the corrected intermediate optical flow map generated by optical flow correction module 712. In some embodiments, frame synthesis step (i.e., operation) 1508 is performed by optical flow correction module 712. Frame synthesis step (i.e., operation) 1508 may use warping, blending, and / or other frame synthesis techniques to synthesize intermediate frames based on the corrected intermediate frames and the start frame 102 and end frame 104 of input frame 702. In some embodiments, the warped frame generated by the warping operation 1104 of optical flow correction module 712 is used as the intermediate frame 714.

[0141] Pixel Correction

[0142] In some embodiments, the pixel correction module 716 corrects the generated intermediate frames 714 based on information obtained from the frame sequence 702, processes all intermediate frames 714 in one forward pass using a pyramid processing structure, and improves temporal consistency. Intermediate frames 714 may still contain artifacts due to inaccurate optical flow or fusion masks.

[0143] like Figure 13 As shown, the pixel correction module 716 adaptively corrects the generated intermediate frames 714 using a temporal pyramid structure similar to that of the optical flow correction module 712. However, due to the misalignment of the generated intermediate frames 714, feeding all the generated intermediate frames 714 to the first level 1352 of the pixel correction module 716 may not adequately improve the quality of the generated intermediate frames. Instead, each generated intermediate frame 714 is provided at different levels of the convolutional neural network according to their temporal distance: I t1 and It7 Provided to Level 1352; I t2 and I t6 Provided to Level 2, 1354; I t3 and I t5 Provided to Level 3 1356; and I t4 Provided to Level 4, 1358. At each timestamp. t i The warped input generated by the warping operation 1104 of the optical flow correction module 712 and It is also fed into each level to reduce errors caused by inaccurate fusion masks. Similar to the optical flow correction module 712, it corrects intermediate frames. It is also fed to the next level to help correct the generated intermediate frame 714, which is closer to the intermediate timestamp.

[0144] like Figure 13 As shown, the same pixel correction subnetwork 718 is used for each level of the pyramid. The pixel correction subnetwork 718 consists of two residual blocks, one convolutional block at the input and the other at the output. The residual blocks can be as described by Nah, S., Hyun Kim, T., and Mu Lee, K. Deep multi-scale convolutional neural networks for deblurring dynamic scenes (Deep multi-scale convolutional neural network for dynamic scene deblurring) The information is described in the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883-3891 (2017) (the full text of which is incorporated herein by reference).

[0145] For the two temporal pyramid correction modules 712 and 716, residual learning can be used to learn optical flow and frame residuals. In optical flow correction module 712, the number of channels can be set in descending order (e.g., ...). Figure 10 The 64, 48, 32, and 24 channels shown are used because fewer frames are processed when moving to the middle time step. In contrast, in the illustrated embodiment, the same number of channels are used for all levels of the pixel correction module 716.

[0146] In operation, the optical flow generated by the optical flow correction module 712 is used to warp the initial image (the start frame 102 and end frame 104 of the input frame 702) and create an intermediate frame 714. The intermediate frame 714 is then passed to the pixel correction module 716 for pixel-level correction using another pyramid processing structure, and a final corrected intermediate frame 720 is generated as the output. Figure 13The architecture of an exemplary pixel correction module 716 is shown. Due to motion offsets between intermediate frames 714 and the use of a rich RGB color space, inputting all seven candidate intermediate frames 714 at the first level 1352 may not provide optimal pixel correction. Earlier levels (e.g., 1352 and 1354) may not have a large receptive field to cover all motion ranges, and the rich image content of the seven frames may clutter the earlier levels; it is easier to accurately correct frames closer to the start frame 102 or end frame 104. Therefore, compared to the optical flow correction module 712, the pixel correction module 716 adjusts the pixel correction based on the distance of the intermediate frame 714 from the start frame 102 and end frame 104 and its distance from the intermediate time step (in this case, ). t 4) The proximity of features is determined by receiving intermediate frames 714 as input at different levels of the neural network. To achieve information sharing, feature maps from earlier levels are passed to later levels.

[0147] At level 1352, cascade operation 1102 receives two intermediate frames. I t1 and I t7 Relevant input data: by The first intermediate frame input data 1302 and the data composed of The seventh intermediate frame input data 1314 is composed of these inputs 1302 and 1314. These inputs are cascaded and fed into the first pixel correction sub-network 718, shown here as a 64-channel sub-network of a pixel correction convolutional neural network. The first pixel correction sub-network 718 is fed through a leakage... relu The convolutional block of function 1310 generates a corrected frame 720 corresponding to its input intermediate frame 714 as output: the first corrected intermediate frame. and the seventh corrected intermediate frame .

[0148] The second level 1354 uses a cascading operation 1102 to concatenate the feature map output of the first pixel correction sub-network 718 with its input, the second intermediate frame input data 1304, and the sixth intermediate frame input data 1312 to generate other corresponding correction frames 720: the second correction intermediate frame. and the sixth corrected intermediate frame .

[0149] The third level 1356 repeats the same process as the second level 1354, with input data for the third intermediate frame 1306 and the fifth intermediate frame 1310, and output of the corrected frame 720 as the third corrected intermediate frame. and the fifth corrected intermediate frame .

[0150] Level 4, 1358, repeats the same process, but uses only a single input, namely the fourth intermediate frame input data 1308, and produces a single corrected frame 720 output, namely the fourth corrected intermediate frame. .

[0151] Training and loss function

[0152] Figure 13 and Figure 14 The basic structure of an adversarial learning scheme for training all or some models (e.g., neural networks) included in the multi-frame interpolation system 700 is also illustrated. During training, input frames 702 from the training video sequence are provided as input to the multi-frame interpolation system 700. These input frames 702 include a non-contiguous start frame 102 and end frame 104. In the training video sequence, multiple actual (ground truth) intermediate frames 1334 are temporally located between the start frame 102 and the end frame 104. Corrected frames 720, generated by the pixel correction module 716 based on the input frames 702 and the set of ground truth frames 1334 obtained from the training video sequence, are used to compute the adversarial loss 1390 using the discriminator network 1380 of a temporally consistent generative adversarial network (GAN). In some embodiments, the multi-frame interpolation system 700 may constitute a single pyramid neural network, progressively guiding the frame generation process at a higher level within the single pyramid network. Temporal consistency can be utilized by jointly optimizing all intermediate frames. Generative adversarial learning schemes can also be used to impose consistency between interpolated frames, as Zhang, H., Shen, C., Li, Y., Cao, Y., Liu, Y., and Yan, Y. Utilizing time Exploiting temporal consistency for real-time video depth estimation depth estimation) The concept of a two-player min-max game can be used to train discriminator networks D, as described in the proceedings of the IEEE International Conference on Computer Vision, pp. 1725-1734 (2019) (the full text of which is incorporated herein by reference). This is based on the work of Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Networks adversarial nets). Advances in Neural Information Processing Systems, pp. 2672-2680 (2014) (the full text of which is incorporated herein by reference). Therefore, GANs (i.e., the parameters of the generator) can optimize the following problems:

[0153] (Equation 13)

[0154] Where g = [ I t1 gt , … I t7 gt ] represents seven ground truth frames, x = [ I –1 , I 0, I 1, I [2] consists of four input frames. The input to the discriminator 1380 is a concatenation of seven corrected intermediate frames 720 generated by the neural network of the pixel correction module 716 of the multi-frame interpolation system 700, used as the generator of the GAN. The following generation components of the GAN are added as time loss:

[0155] (Equation 14)

[0156] To optimize the optical flow correction module 712 and the pixel correction module 716, the following applications are used. Loss. The entire multi-frame interpolation system is trained by combining all loss functions.

[0157] (Equation 15)

[0158] Where λ is the weighting coefficient, which is equal to 0.001.

[0159] The training example of the model (or neural network) included in the multi-frame interpolation system 710 is now described. To train the neural network included in the multi-frame interpolation system 700, a training dataset of 240 fps video clips with a resolution of 720 × 1280 pixels was assembled. To cover various scene types, 903 short videos with lengths between 2 and 10 seconds were selected. 8463 samples were created from these videos, each consisting of 25 consecutive frames. Each sample was used as a batch of training data, so each batch included 25 consecutive sample frames. The multi-frame interpolation system 700 received frames 1, 9, 17, and 25 as input and generated seven frames between frames 9 and 17 by treating frames 10 through 16 as its ground truth. Randomly modified versions of the samples were also used for data augmentation during training: 352 × 325 pixel pieces were randomly cropped from the sample frames, and horizontal, vertical, and time flips were applied.

[0160] To improve convergence speed, the neural networks included in the multi-frame video frame interpolation software system 700 are trained in stages. First, each neural network, except the discriminator network, is trained without updating the parameters of other networks, at a speed of 10... –4 The learning rate is used independently over 15 cycles. Training is performed using a loss of 1. Then, equation (15) and 10 are used as above. –5 The learning rate was used to jointly train the neural network included in the multi-frame video frame interpolation software system 700 within 100 epochs. Adam optimizers (such as Kingma, DP, and Ba, J.) were used. Adam: A method for stochastic optimization optimization) (as described in arXiv preprint, arXiv:1412.6980 (2014)), the neighborhood range in equation (4) d Set to 9. During the training of the neural network included in the multi-frame video frame interpolation software system 700, the pixel values ​​of all images are expanded to the range [–1, 1]. Training can be performed on an Nvidia P100 GPU.

[0161] Other applications

[0162] A multi-frame interpolation system 700 has already been described for applications in the field of slow-motion video generation. However, some embodiments described herein can be used for a wide range of applications, such as video in-painting, video object removal, and high-resolution 3D reconstruction of objects from video clips. The general temporal pyramid structure described in the context of optical flow correction module 712 and pixel correction module 716 can be used for a range of sequence processing and interpolation tasks, wherein the quality of the output data produced by the task is closer to the given input data provided to the task than that of the output data further away from the input data. The input data can be a time series or any other sequence: for example, business analytics data, curve super-resolution or high-resolution construction and analysis.

[0163] In some embodiments, the motion modeling methods described in the context of the higher-order motion modeling module 708 can be used to improve the performance of motion estimation and synthesis in, for example, computer vision and computer graphics. An exemplary application is temporally resolution-limited motion capture, using higher-order motion modeling to generate ultra-high-resolution video frames with minute motion details.

[0164] It should be understood that the methods, devices, systems and media described herein are also suitable for other practical applications.

[0165] Method and processor readable medium

[0166] The steps and / or operations in the flowcharts and accompanying drawings described herein are for illustrative purposes only. These steps and / or operations can be varied in many ways without departing from the teachings of this invention. For example, steps may be performed in a different order, or steps may be added, deleted, or modified.

[0167] In light of the present invention, the software code used to perform the described methods is within the scope of those skilled in the art. Machine-readable code that can be executed by one or more processors of one or more corresponding devices to perform the described methods can be stored in a machine-readable medium such as the memory of a data manager. The terms "software" and "firmware" are interchangeable in this invention and include any computer program stored in memory for execution by a processor, including random access memory (RAM), read-only memory (ROM), EPROM, electrically EPROM (EEPROM), and non-volatile RAM (NVRAM). The above memory types are merely examples and therefore do not limit the types of memory that can be used to store computer programs.

[0168] Overview

[0169] All values ​​and subranges within the scope of disclosure are also disclosed. Furthermore, while the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements, these systems, devices, and components may be modified to include more or fewer such elements. Although several exemplary embodiments are described herein, modifications, adaptations, and other implementations may be made. For example, elements shown in the accompanying drawings may be replaced, added, or modified, and the exemplary methods described herein may be modified by replacing, reordering, or adding steps of the disclosed methods. In addition, numerous specific details are set forth to provide a thorough understanding of the exemplary embodiments described herein. However, those skilled in the art will understand that the exemplary embodiments described herein may be practiced without these specific details. Furthermore, well-known methods, processes, and elements have not been described in detail so as not to obscure the exemplary embodiments described herein. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

[0170] Although the invention has been described, it will be understood, at least in part, by those skilled in the art, that the invention also relates to various elements for performing at least some aspects and features of the described methods by means of hardware, software, or a combination thereof. Therefore, the technical solutions of the invention can be embodied in non-volatile or non-transitory machine-readable media (e.g., optical discs, flash memory, etc.) on which executable instructions tangibly stored are stored, enabling a processing device to perform examples of the methods disclosed herein.

[0171] The term "processor" can include any programmable system, including systems using microprocessors / controllers or nanoprocessors / controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), reduced instruction set circuits (RISCs), logic circuits, and any other circuits or processors capable of performing the functions described herein. The term "database" can refer to a data volume, a relational database management system (RDBMS), or both. As used herein, a database can include any collection of data, including hierarchical databases, relational databases, flat file databases, object-relational databases, object-oriented databases, and any other structured records or collections of data stored in a computer system. The examples above are merely illustrative and are not intended to limit the definition and / or meaning of the terms "processor" or "database" in any way.

[0172] This invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are to be regarded in all respects as illustrative rather than restrictive. This invention is intended to cover and encompass all suitable technical changes. Therefore, the scope of this invention is described by the appended claims rather than by the foregoing description. The scope of the claims should not be limited to the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the entire description.

[0173] Further details of the exemplary embodiments, experimental results, comparisons with other techniques, identification of data used to train and test the embodiments, and other references relied upon are set forth in the following paper entitled "Temporally Adaptive Multiframe Interpolation with Higher-Order Motion Modeling," which will be considered an integral part of this detailed description. All external references cited herein are incorporated herein by reference in their entirety.

Claims

1. A method for interpolating multiple video frames, characterized in that, The method includes: Obtain multiple optical flow maps corresponding to multiple times between the start and end times; The plurality of optical flow maps are processed to generate a first corrected optical flow map corresponding to a first intermediate time close to the start time or the end time; The plurality of optical flow maps and the first corrected intermediate optical flow map are processed to generate a second corrected optical flow map corresponding to a second intermediate time that is far from the start time and the end time; A first new intermediate frame is generated based on the first corrected optical flow map; A second new intermediate frame is generated based on the second corrected optical flow map.

2. The method according to claim 1, characterized in that, Also includes: The received video sequence includes: The starting frame of the video sequence, the starting frame corresponding to the start time of the video sequence; The end frame of the video sequence, the end frame corresponding to the end time of the video sequence; Receive two additional frames of the video corresponding to two additional times, each additional time being before the start time of the video sequence or after the end time; Generate at least one initial optical flow map corresponding to the estimated optical flow between the start frame and the end frame; By applying a motion model to the at least one initial optical flow graph, the at least one initial optical flow graph is processed to generate a plurality of intermediate optical flow graphs, the plurality of intermediate optical flow graphs corresponding to a plurality of intermediate times between the start time and the end time of the video sequence.

3. The method according to claim 2, characterized in that, The motion model is of third order or higher.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: The plurality of optical flow maps are processed to generate at least one additional corrected optical flow map, wherein the first corrected optical flow map, the second corrected optical flow map, and the at least one additional corrected optical flow map constitute a plurality of corrected optical flow maps corresponding to the plurality of times; At least one additional new intermediate frame is generated based on the at least one additional modified optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times.

5. The method according to claim 4, characterized in that, Also includes: Obtain the multiple new intermediate frames; The first new intermediate frame and the second new intermediate frame are corrected in the following way: Based on the plurality of new intermediate frames, a first correction frame corresponding to a first time close to the start time or the end time is generated; Based on the plurality of new intermediate frames and the first correction frame, a second correction frame corresponding to a second time that is far from the start time and the end time is generated.

6. The method according to claim 5, characterized in that, It also includes correcting the plurality of new intermediate frames to generate a plurality of corrected new intermediate frames based on the plurality of new intermediate frames.

7. The method according to claim 2, characterized in that, Also includes: The plurality of intermediate optical flow maps are generated using an optical flow model trained using a supervised learning algorithm, a training dataset, and a relaxation loss function configured to allow a predetermined amount of error in the estimated optical flow during the training of the optical flow correction model.

8. The method according to claim 7, characterized in that, The relaxation loss function includes an adversarial loss function generated by a generative adversarial network.

9. The method according to claim 4, characterized in that, The at least one additional modified optical flow map is generated using a pixel modification model, which is trained using an adversarial loss function generated by a generative adversarial network.

10. The method according to claim 1, characterized in that, Processing the plurality of optical flow maps to generate a first corrected optical flow map and processing the plurality of optical flow maps and the first corrected intermediate optical flow map to generate a second corrected optical flow map are performed using an optical flow correction model trained using an adversarial loss function generated by a generative adversarial network.

11. A computing system, characterized in that, include: processor; A memory storing processor-executable instructions, wherein, when executed by the processor, the computing system: Obtain multiple optical flow maps corresponding to multiple times between the start and end times; The plurality of optical flow maps are processed to generate a first corrected optical flow map corresponding to a first intermediate time close to the start time or the end time; The plurality of optical flow maps and the first modified optical flow map are processed to generate a second modified optical flow map corresponding to a second intermediate time that is far from the start time and the end time; A first new intermediate frame is generated based on the first corrected optical flow map; A second new intermediate frame is generated based on the second corrected optical flow map.

12. The computing system according to claim 11, characterized in that, When the instructions are executed by the processor, the computing system is also affected: The received video sequence includes: The starting frame of the video sequence, the starting frame corresponding to the start time of the video sequence; The end frame of the video sequence, the end frame corresponding to the end time of the video sequence; Receive two additional frames of the video corresponding to two additional times, each additional time being before the start time of the video sequence or after the end time; Generate at least one initial optical flow map corresponding to the estimated optical flow between the start frame and the end frame; By applying a motion model to the at least one initial optical flow graph, the at least one initial optical flow graph is processed to generate a plurality of intermediate optical flow graphs, the plurality of intermediate optical flow graphs corresponding to a plurality of intermediate times between the start time and the end time of the video sequence.

13. The computing system according to claim 12, characterized in that, The motion model is of third order or higher.

14. The computing system according to any one of claims 11 to 13, characterized in that, When the instructions are executed by the processor, the computing system is also affected: The plurality of optical flow maps are processed to generate at least one additional corrected optical flow map, wherein the first corrected optical flow map, the second corrected optical flow map, and the at least one additional corrected optical flow map constitute a plurality of corrected optical flow maps corresponding to the plurality of times; At least one additional new intermediate frame is generated based on the at least one additional modified optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times.

15. The computing system according to claim 14, characterized in that, When the instructions are executed by the processor, the computing system is also affected: Obtain the multiple new intermediate frames; The first new intermediate frame and the second new intermediate frame are corrected in the following way: Based on the plurality of new intermediate frames, a first correction frame corresponding to a first time close to the start time or the end time is generated; Based on the plurality of new intermediate frames and the first correction frame, a second correction frame corresponding to a second time that is far from the start time and the end time is generated.

16. A multi-frame video frame interpolation system, characterized in that, include: Optical flow correction module, used for: Obtain multiple optical flow maps corresponding to multiple times between the start and end times; The plurality of optical flow maps are processed to generate a first corrected optical flow map corresponding to a first intermediate time close to the start time or the end time; The plurality of optical flow maps and the first corrected intermediate optical flow map are processed to generate a second corrected optical flow map corresponding to a second intermediate time that is far from the start time and the end time; A first new intermediate frame is generated based on the first corrected optical flow map; A second new intermediate frame is generated based on the second corrected optical flow map.

17. The multi-frame video frame interpolation system according to claim 16, characterized in that, Also includes: The optical flow estimation module is used for: The received video sequence includes: The starting frame of the video sequence, the starting frame corresponding to the start time of the video sequence; The end frame of the video sequence, the end frame corresponding to the end time of the video sequence; Receive two additional frames of the video corresponding to two additional times, each additional time being before the start time of the video sequence or after the end time; Generate at least one initial optical flow map corresponding to the estimated optical flow between the start frame and the end frame; A motion modeling module is used to process the at least one initial optical flow graph by applying a motion model to the at least one initial optical flow graph to generate a plurality of intermediate optical flow graphs, the plurality of intermediate optical flow graphs corresponding to a plurality of intermediate times between the start time and the end time of the video sequence.

18. The multi-frame video frame interpolation system according to claim 17, characterized in that, The motion modeling module is used to generate the plurality of intermediate optical flow maps using a motion model of third order or higher.

19. The multi-frame video frame interpolation system according to any one of claims 16 to 18, characterized in that, The optical flow correction module is also used for: The plurality of optical flow maps are processed to generate at least one additional corrected optical flow map, wherein the first corrected optical flow map, the second corrected optical flow map, and the at least one additional corrected optical flow map constitute a plurality of corrected optical flow maps corresponding to the plurality of times; At least one additional new intermediate frame is generated based on the at least one additional modified optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times.

20. The multi-frame video frame interpolation system according to claim 19, characterized in that, It also includes a pixel correction module, which is used for: Obtain the multiple new intermediate frames; The first new intermediate frame and the second new intermediate frame are corrected in the following way: Based on the plurality of new intermediate frames, a first correction frame corresponding to a first time close to the start time or the end time is generated; Based on the plurality of new intermediate frames and the first correction frame, a second correction frame corresponding to a second time that is far from the start time and the end time is generated.

21. A non-transitory computer-readable medium that tangibly stores instructions, characterized in that, When the instruction is executed by the processing unit, it causes the processing unit to perform the method according to any one of claims 1 to 10.

22. A computer program comprising instructions, characterized in that, When the instruction is executed by the processing unit, it causes the processing unit to perform the method according to any one of claims 1 to 10.

Images