MULTI-SMART VIDEO INTERPOLATION WITH OPTICAL FLOW

Abstract

A computer-implemented method comprising: receiving forward optical flow data (F0→1) calculated from a first frame (I0) to a second frame (I1) in a sequence of frames, and receiving forward optical intermediate flow data (F̂t→0) for a time in the sequence of frames that lies between the first frame and the second frame; receiving backward optical flow data calculated from the second frame to the first frame, and receiving backward optical intermediate flow data (F̂t→1) for the time;and a generation, by a neural network model for flow interpolation (102, 122), of an intermediate frame (It) at time based on the first frame (I0) and the second frame (I1), the forward optical intermediate flow data (F̂t→0), the backward optical intermediate flow data (F̂t→1), the first frame distorted in accordance with the forward optical intermediate flow data (F̂t→0) (I0→t), and the second frame distorted in accordance with the backward optical intermediate flow data (F̂t→1) (I1→t), further comprising an interpolation, by the neural network model for flow interpolation (102, 122), between the forward optical intermediate flow data (F0→1) and the backward optical intermediate flow data to obtain the forward optical intermediate flow data (F̂t→0) and the to generate backward-directed optical intermediate flow data (F̂t→1) at that time.;

Description

FIELD OF THE INVENTIONThe invention relates to video interpolation and more particularly to video interpolation using an optical flow.BACKGROUNDVideo interpolation is used to generate one or more additional frames between two successive frames. Increasing the number of frames enables video reproduction with a higher frame rate and slow motion reproduction. Video interpolation can also be used to generate smooth image transitions. The intermediate individual images produced should be both spatially and temporally coherent, so that artifacts caused by occlusion are minimized. Conventional techniques produce a single frame between two successive frames so that when large movements occur between the two successive frames, artifacts are present in the frame.The conventional techniques cannot be used directly to generate videos with arbitrarily higher frame rate. One approach to simply recursively applying a frame video interpolation method to generate multiple intermediate frames is subject to at least two constraints. First, recursive frame interpolation cannot be completely parallelized and is therefore slow because some frames cannot be computed until other frames are complete (e.g., in seven frame interpolation, frame 2 depends on 0 and 4, while frame 4 depends on 0 and 8). Furthermore, errors accumulate during recursive interpolation. Second, frame interpolation can only generate 2n-1 intermediate frames (e.g., 3, 7). As a result, recursively applying frame video interpolation to generate 1008 fps (frames per second) video from 24 fps video requires the generation of 41 intermediate frames which is not efficient.Recently, optical flow techniques have been used to perform frame interpolation. However, motion boundaries and strong occlusions are still a challenge, such that the interpolated frames tend to have artifacts in the vicinity of the boundaries of moving objects. There is a need to treat these questions and / or other questions associated with the prior art.Liu, Ziwei, et al., "Video frame synthesis using deep voxel flow." Proceedings of the IEEE international conference on computer vision. 2017, combines optical flow techniques with neural networks for the generation of new frames to reduce image artifacts in video interpolation.It is an object of the present invention to effectively increase a frame rate in a video sequence with little artifacts.BRIEF DESCRIPTIONThis object is achieved by the features of the independent claims. Preferred embodiments are described in the dependent claims.A method according to claim 1, a computer readable storage medium according to claim 18 and an optical flow video interpolation system according to claim 13 are provided. Forward optical flow data calculated from a first frame to a second frame in a sequence of frames and intermediate forward optical flow data for a time in the sequence of frames that is between the first frame and the second frame are received. Backward optical flow data calculated from the second frame toward the first frame and intermediate backward optical flow data for time are received. A neural network model for flow interpolation generates an intermediate image at this time based on the first frame and the second frame, the forward optical intermediate flow data, the backward optical intermediate flow data, the first frame distorted in accordance with the forward optical intermediate flow data, and the second frame distorted in accordance with the backward optical intermediate flow data. Interpolating, by the flow interpolating neural network model, between the forward optical flow data and the backward optical flow data to generate the forward optical intermediate flow data and the backward optical intermediate flow data at the time.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1A illustrates a block diagram of a frame interpolation system according to an embodiment. FIG. 1B is a conceptual diagram of flow interpolation, according to an embodiment. FIG. 1C illustrates a flow diagram of a method for generating an intermediate frame via flow interpolation, according to an embodiment. FIG. 1D is a conceptual diagram of forward and backward warping, according to an embodiment. FIG. 1E illustrates a block diagram of another frame interpolation system according to an embodiment. FIG. 1F illustrates a flow diagram of another method for generating an intermediate frame via flow interpolation, according to an embodiment. FIG. 2A illustrates a block diagram of another frame interpolation system according to an embodiment. FIG. 2B illustrates a flow diagram of a method for training the frame interpolation system according to an embodiment. FIG. 2C illustrates input frames, bidirectional optical flows, and distorted input frames, according to an embodiment. FIG. 2D illustrates visibility maps and predicted intermediate frames, according to an embodiment. FIG. 3 illustrates a parallel processing unit according to an embodiment. FIG. 4A illustrates a general processing cluster within the parallel processing unit of FIG. 3, according to an embodiment. FIG. 4B illustrates a memory partitioning unit of the parallel processing unit of FIG. 3 according to an embodiment. FIG. 5A illustrates the streaming multiprocessor of FIG. 4A, according to an embodiment. FIG. 5B is a conceptual diagram of a processing system implemented using the PPU of FIG. 3, according to one embodiment. FIG. 5C illustrates an example system in which the various architecture and / or functionality of the various previous embodiments may be implemented.DETAILED DESCRIPTIONThere are many unfriendly moments in life one would like to capture with a camera in slow motion because the details are difficult to see with the eyes: the first time a baby runs, a difficult skateboard trick, a dog capturing a ball, etc. While it is possible to capture videos at a speed of 240 fps (frames per second) with a mobile phone, high speed professional cameras are still needed for higher frame rates. Moreover, many of the moments we wish to slow down are unpredictable and are therefore recorded at standard frame rates. Recording everything at high frame rates is impractical - it requires large storage capacity and is energy intensive for mobile devices. Therefore, it is of great interest to generate high quality slow motion video from existing videos. In addition to converting standard video to higher frame rates, video interpolation can be used to generate smooth frame transitions. Video interpolation also has interesting new applications in self-supervised learning and serves as a supervisory signal to learn optical flow from unsolicited videos.A neural network model may be trained to interpolate one or more intervening (intermediate) neural networks for a video sequence. For example, 240 frames per second videos may be generated from standard sequences (30 frames per second) by interpolating seven intermediate frames for each two consecutive frames. The intermediate frames generated by the neural network model are both spatially and temporally coherent within the video sequence. A successful solution should not only correctly interpret the movement between two input images (implicitly or explicitly) but also understand occlusions in order to reduce artifacts in the interpolated individual images, in particular around movement boundaries.FIG. 1A illustrates a block diagram of a frame interpolation system 100 according to an embodiment. A sequential pair of images of color data (I 0 and I 1) is input to the frame interpolation system 100. Generally, the frame interpolation system 100 receives input data at a time t and at a time t+1, such as two frames of a sequence of frames. The input frame sequence may include dynamic scenes captured by a stationary or moving (e.g., panning) camera.The frame interpolation system 100 includes an optical flow distortion unit 105, a flow refinement neural network model 110, and a flow interpolation neural network model for each of the input sequential frames I 0 and I 1. The frame interpolation system 100 also includes a frame prediction unit 115 configured to predict the intermediate frame I t at a time t ∈(0,1).Although the frame interpolation system 100 is described in the context of processing units, one or more of the optical flow warping unit 105, the neural network model for flow interpolation 102, and the image prediction unit 115 may be executed by a program, a user-defined circuit, or by a combination of a user-defined circuit and a program. For example, the flow interpolation neural network model 102 may be implemented by a GPU (graphics processor unit), a CPU (central processor unit), or any processor that may implement layers of a neural network. Further, it will be understood by those skilled in the art that any system that performs the operations of the frame interpolation system 100 is within the scope and scope of embodiments of the invention.The frame interpolation system 100 is trained to directly output the RGB pixels of I t. Importantly, the frame interpolation system 100 must learn to interpret not only the motion patterns but also the appearance of the two input images. Because of the powerful RGB color space, accurate prediction of high quality intermediate frames is difficult.Bidirectional optical flows (F 1→0, F 0→1) are calculated based on the input frames (I 0, I 1). The bidirectional optical flows are linearly combined to approximate bidirectional optical intermediate flows (F̂ t→0, F̂ t→1) for at least one time step t between the two input frames. The input frames are respectively distorted by the optical flow distortion units 105- 0 and 105- 1 in accordance with the approximated bidirectional optical intermediate flows for each time step to generate distorted input frames 0→t and 1→t.The flow refinement neural network model 102 includes flow refinement neural networks 110- 0 and 110- 1 and an image prediction unit 115. The bidirectional optical intermediate flows (F̂ t→0, F̂ t→1) are refined for each time step by the neural network model for flow refinement 110 using the two input frames, the bidirectional optical intermediate flows and the two distorted input frames. The refined bi-directional optical flows (F t→0, F t→1) are output from the neural network models 110- 0 and 110- 1. The refined bi-directional optical intermediate flows are processed by the image prediction unit 115 to generate the intermediate frame I t.In one embodiment, the image prediction unit 115 receives the two input frames and distorts the two input frames (backwards) according to the refined bi-directional optical intermediate flows for each time step to generate refined distorted input frames. Therefore, additional optical flow warping units 105 may be included in the image prediction unit 115 to generate the refined distorted input frames, and the refined distorted input frames are effectively linearly merged by the image prediction unit 115 to generate the intermediate frame for each time step. In one embodiment, the image prediction unit receives the distorted input images (Lithographically 0→t, Lithographically 1→t), generated by the optical flow distortion units 105- 0 and 105- 1, and the distorted input frames are effectively linearly merged by the image prediction unit 115 to generate the intermediate frame for each time step.Further illustrative information is provided below about various optional architectures and features with which the aforementioned framework may be implemented according to the desires of the user. It is to be expressly understood that the following information is illustrative and not to be construed as limiting. Any of the following features may optionally be incorporated with or without the exclusion of other described features.FIG. 1B is a conceptual diagram of intermediate interpolation for optical flow, in accordance with an embodiment. At a time T=0, a set of pixels (filled circles) indicates the first frame I 0, shown in a first column, and at a time T=1, the set of pixels indicates the second frame I 1, shown in a third column. At time T=t, a value of the center pixel (open circle) is approximated, and the optical flow data of pixels at the same position in the first and second frames (at T=0 and T=1) represented as the solid arrows are lent out. The rental out of the optical flow data of pixels in the first and second frames provides an accurate result when the optical flow field is locally smooth.As illustrated in FIG. 1B, the bidirectional optical flow (F 0→1) from the input frame I 0 to the input frame I 1 is calculated, and the bidirectional optical flow (F 1→0) from the input frame I 1 to the input frame I 0 is calculated. The bidirectional optical flows are linearly combined to approximate intermediate bidirectional optical flows (F̂ t→0, F̂ t→1) for time step t between the two input frames, shown as dashed arrows. In particular, a forward optical intermediate flow F t→1( p) may be approximated as or wherein the direction of the optical flow between the two input frames is taken in the same or in the opposite direction and the magnitude of the bidirectional optical flow is scaled accordingly ((1-t) in equation (2)). Similar to the temporal consistency for RGB image synthesis, in one embodiment, the intermediate optical flow may be synthesized by combining the bidirectional optical flow as follows (in vector form).Approximating the intermediate optical flow by combining the bi-directional optical flow works well in smooth regions, but poorly around motion boundaries because the motion is locally non-smooth near motion boundaries. To reduce artifacts around motion boundaries that may result in poor image prediction, the flow refinement neural network model 110 learns to refine the bi-directional optical intermediate flows received from the flow interpolation neural network model 102. In one embodiment, the flow interpolation neural network model 102 is a convolutional neural network (CNN). The flow interpolation neural network model 102 refines the intermediate bidirectional optical flows (F̂ t→0, F̂t→1) for each time step using the intermediate bidirectional optical flows (F̂ t→0 F̂ t→1), of the two input frames and the input frames distorted by the optical flow distortion units 105.FIG. 1C illustrates a flow diagram of a method 120 for generating an intermediate frame via flow interpolation, according to an embodiment. Although the method 120 is described in the context of a processing unit, the method 120 may also be performed by a program, a user-defined circuit, or by a combination of a user-defined circuit and a program. For example, the method 120 may be performed by a GPU (graphics processor unit), a CPU (central processor unit), or any processor capable of implementing a neural network model. Moreover, it will be understood by those skilled in the art that any system that performs method 120 is within the scope and spirit of embodiments of the invention.In step 125, the optical flow data (F 0→1) and the intermediate optical flow data (F̂ t→0) are received from a flow refinement neural network model 110- 0. In one embodiment, the forward optical flow data (F 0→1) is calculated from a first frame I 0 to a second frame I 1 in a sequence of frames. In an embodiment, the backward optical flow data is calculated from the second frame I 1 to the first frame I 0 and the forward intermediate optical flow data and the backward intermediate optical flow data are predicted using the forward and backward optical flow data. In one embodiment, the first frame I 0 and the second frame I 1 are processed by an intermediate flow neural network model to generate the forward optical flow data and the backward optical flow data.The forward intermediate optical flow data applies for a time T=t in the sequence of frames that lies between the first frame and the second frame. In an embodiment, color data for each frame may be represented as red, green, and blue (RGB) color components, YUV components, or the like. Optical flow data includes a flow field that indicates a direction and magnitude of image data movement from the first frame to the second frame. In other words, a particular color of a particular pixel or an edge of an object in the first frame may move to another pixel in the second frame. The color may be associated with an object in a scene moving between the first and second frames. Alternatively, a camera position used to capture the scene may change (pan) between the first and second frames so that the object appears to move.In step 130, intermediate backward optical flow data (F̂ t→1) is received from the neural network model for flow refinement 110- 1. The backward optical intermediate flow data (F̂ t→1) is in the sequence of frames that lies between the first frame and the second frame for the time T=t.In step 135, the flow interpolation neural network model 102 generates an intermediate frame at this time based on the first frame and the second frame, the forward intermediate optical flow data, the backward intermediate optical flow data, the first frame distorted in accordance with the forward intermediate optical flow data 0→t), and the second frame distorted in accordance with the backward intermediate optical flow data ( . In one embodiment, the optical flow distortion unit 105- 0 calculates the first frame that has been distorted in accordance with the forward intermediate optical flow data. In one embodiment, the optical flow distortion unit 105- 1 calculates the second frame that has been distorted in accordance with the backward optical interflow data.FIG. 1D is a conceptual diagram of forward and backward warping, according to an embodiment. The variable length, high quality multi-frame interpolation technique performed by the frame interpolation system 100 predicts a frame at any time step between two frames by warping the input two images to the specific time step and then adaptively merging the two distorted images to generate the intermediate frame. In one embodiment, instead of performing a forward warping operation, a backward warping operation is applied to the first and second input images. For the backward distortion, each pixel in the intermediate (target) frame finds a correspondence in the input image (source image). When the forward distortion is used, pixels 126 in the target image do not have corresponding pixels in the source image and pixels 136 correspond to multiple pixels in the source image. Therefore, backward warping can produce more accurate distorted image data.If the two intermediate bidirectional optical flows (F̂t→0, F̂t→1) are known, the intermediate frame I t can be synthesized as follows: wherein g(...) is a backward distortion function. The backward distortion function may be implemented using bilinear interpolation and is differentiable, thereby enabling training of the frame interpolation system 100. The parameter α 0 controls the contribution of the two distorted images and depends on two factors: the temporal consistency and the occlusion argument. denotes an element-by-element multiplication and implies a content-aware weighting of input frames. For the temporal consistency, the closer the time step T=t is to T=0, the more I 0 contributes to Characterization t and a similar property applies to I 1.FIG. 1E illustrates a block diagram of another frame interpolation system 150 according to an embodiment. The frame interpolation system 150 includes a flow interpolation neural network model 122, an intermediate optical flow neural network model 142, and optical flow distortion units 105. The intermediate optical flow neural network model 142 receives the two input frames and approximates the intermediate bidirectional optical flows F̂ t→0 and F̂ t→1. In one embodiment, the intermediate optical flow neural network model 142 is a CNN. The flow interpolation neural network model 122 includes a flow refinement neural network model 145- 0, a flow refinement neural network model 145- 1, and an image prediction unit 155.Although the frame interpolation system 150 is described in the context of processing units, one or more of the intermediate flow neural networks 142, the optical flow distortion units 105, the flow refinement neural networks 145, and the image prediction unit 155 may be executed by a program, a user-defined circuit, or by a combination of a user-defined circuit and a program. For example, the neural network models for flow refinement 145 may be implemented by a GPU (graphics processor unit), a CPU (central processor unit), or any processor capable of implementing layers of a neural network. Moreover, it will be understood by those skilled in the art that any system that performs the operations of the frame interpolation system 150 is within the scope and scope of the invention.An important property of the problem of interpolating video frames is that when a pixel p is visible at T = t, it is most likely visible in at least one of the input frames, which means that occlusion can be handled. To account for occlusion, the refinement neural network models 145- 0 and 145- 1 predict soft visibility maps V t←0 and V t←1. for each time step. V t←0( p) ∈[0,1]indicates whether the pixel p remains visible (0 is fully occluded and therefore not visible) upon the change from T=0 to T=t. The two visibility maps must satisfy the following limitationWithout such limitation, the refined neural network models 145 may diverge during training. V t←0( p)=0 intuitively implies V t←1( p)=1, which means that the pixel p of I 0 is occluded at T=t, so that I 1 is to be used for prediction of the intermediate frame. On the other hand, when V t←0( p)=1V t←1( p)=0 implies, meaning that the pixel p of I 1 is occluded at T=t, I 0 is to be used to predict the intermediate frame. Note that it is rare that a pixel at time t is occluded at both time T=0 and time T=1. Since soft visibility maps are used when pixel p is visible in both I 0 and I 1 the neural network model for flow interpolation 122 learns to adaptively combine the information from two images.The soft visibility maps are applied to the input distorted frames before the input distorted frames are linearly merged by the image prediction unit 155 to generate the intermediate frame for each time step. Combining the temporal consistency and occlusion argument allows the interpolated frame to be computed as: where Z = (1 - t)V t→0+ tV t→1 is a normalization factor.In one embodiment, somewhat better performance is achieved by predicting residues of the optical intermediate flow rather than directly predicting the optical intermediate flow in the neural networks for refinement 145. In one embodiment, the neural network models predict t→0 and 145-1F for refinement 145-0 and 145-1, respectively. F t→1. F t→0 and F t→1 are then calculated as:By applying the visibility maps to the distorted images prior to fusion, the contribution of occluded pixels from the interpolated intermediate frame is excluded, thereby avoiding or reducing artifacts. Since none of the parameters (e.g., weights) of the frame interpolation system 150 learned during training are time dependent, the frame interpolation system 150 may generate as many intermediate frames as necessary. In one embodiment, the frame interpolation system 150 may generate as many intermediate frames in parallel as necessary.In one embodiment, the intermediate optical flow neural network model 142 is implemented as an encoder and the flow interpolation neural network model 122 is implemented as a decoder. In one embodiment, the intermediate optical flow neural network model 142 is structured with 6 hierarchies each consisting of two convolutional layers and a leaky rectified linear unit (ReLU) layer. In one embodiment, at the end of each hierarchy, except the last, an average pooling layer with a step size of 2 is used to reduce the spatial dimension. In one embodiment, the flow interpolation neural network 122 is structured with 5 hierarchies. In one embodiment, a bilinear upsampling layer is used at the beginning of each hierarchy to increase the spatial dimension by a factor of 2, followed by two convolutional and leaky ReLU layers.In one embodiment, the frame interpolation system 150 is trained with 1,132,240 fps video clips containing 300K individual video frames. Experimental results on multiple data sets predicting a different number of interpolated frames show that the frame interpolation system 150 consistently cuts better than conventional interpolated imaging techniques. Unlike conventional frame interpolation techniques for performing frame interpolation, the frame interpolation system 150 may perform variable length, multiple frame video interpolation, with motion interpretation and occlusion argument modeled together. The motion interpretation and occlusion argument are modeled in the continuously trainable frame interpolation system 150 including the intermediate optical flow neural network model 142 and the flow enhancement neural network models 145-0 and 145-1.FIG. 1F illustrates a flow diagram of another method 160 for generating an intermediate frame via flow interpolation, according to an embodiment. Although the method 160 is described in the context of a processing unit, the method 160 may also be performed by a program, a user-defined circuit, or by a combination of a user-defined circuit and a program. For example, the method 160 may be performed by a GPU (graphics processor unit), a CPU (central processor unit), or any processor that may implement a neural network model. Moreover, it will be understood by those skilled in the art that any system that performs method 160 is within the scope and scope of embodiments of the invention.In step 162, bidirectional (forward and backward) optical flow data (F 0→1 and F 1→0) are generated. In step 165, bidirectional optical interflow data F̂ t→0 and F̂t→1 are approximated by the neural network model for optical interflow 142. In one embodiment, the flow refinement neural network model 145- 0 calculates F t→0 and then calculates the refined intermediate optical flow data F t→0. In one embodiment, the image prediction unit 115 receives ΔF t→0 and then calculates the refined intermediate optical flow data F t→0. In one embodiment, the neural network flow refinement model 145- 1 calculates ΔF t→1 and then calculates the refined intermediate optical flow data F t→1. In one embodiment, the image prediction unit 115 receives ΔF t→1 and then calculates the refined intermediate optical flow data F t→1.In step 170, the optical flow distortion units 105 distort the input frames using the intermediate bidirectional optical flow data. In step 175, the flow refinement neural network models 145 refine the intermediate bidirectional optical flow data based on the distorted input frames and the input frames. In step 185, the visibility maps are predicted by the flow interpolation neural network model 122 based on the inputs to the flow interpolation neural network model 122. In one embodiment, the flow refinement neural network model 145- 0 predicts the visibility map V t→0 based on the first input frame I 0, the forward intermediate optical flow data F̂ t→0 and the first frame distorted in accordance with the forward intermediate optical flow data (Ull 0→t). In one embodiment, the flow refinement neural network model 145- 1 predicts the visibility map V t→1 based on the second input frame I 1, the backward intermediate optical flow data F̂ t→1 and the second frame distorted in accordance with the backward intermediate optical flow data (Ull 1→t).In step 190, the intermediate frame is predicted by the neural network model for flow interpolation 122. In one embodiment, the image prediction unit 155 predicts the intermediate frame using equation (4) or (6). In an embodiment, in step 190, the image prediction unit 155 applies the visibility maps to the first distorted frame and the distorted second frame before calculating the intermediate frame. In an embodiment, the image prediction unit 155 generates the first distorted frame and the distorted second frame using the intermediate flow refined optical data.FIG. 2A illustrates a block diagram of another frame interpolation system 200 according to an embodiment. In addition to the intermediate flow neural network model 122, the optical flow distortion units 105, and the flow interpolation neural network model 102 or 122, the frame interpolation system 100 includes a training loss function 210 and a flow calculation neural network model 215. In one embodiment, the neural network model for flow calculation 215 is implemented as a CNN. In one embodiment, the neural network model for flow calculation 215 is implemented as an encoder and decoder and has large filters in the first layers of the encoder to detect long-term movements. In one embodiment, 7×7 cores are used in the first two convolutional layers, and 5<5 cores are used in the second hierarchy. In one embodiment, 3×3 convolution kernels are used for layers in the remainder of the neural network model for flow calculation 215.The neural network flow calculation model 215 receives the input frames and uniquely generates the bidirectional optical flow data (forward optical flow data and backward optical flow data) to predict one or more interpolated frames between the input frames. On the other hand, the other components in the frame interpolation system 200, that is, the intermediate optical flow neural network model 122, the optical flow distortion units 105, and the flow interpolation neural network model 102 or 122, and the training loss function 210 calculate data for each time step corresponding to an interpolated frame.During training, the training loss function 210 compares the intermediate frame to a ground truth frame (GR) and updates parameters of the flow calculation neural network model 215, the intermediate flow neural network model 142, and the flow interpolation neural network model 102 or 122 to reduce differences between the intermediate frame and the ground truth frame. Importantly, the flow calculation neural network model 215, the intermediate optical flow neural network model 122, and the flow interpolation neural network model 102 or 122 are trained together to learn the optical flow in an unsupervised manner. Training is performed using video sequences as training datasets, with a subset of the frames being used as exemplary intermediate frames. For example, for a sequence of three frames in the video sequence, the first and third frames may be input to the frame interpolation system 100, and the second frame occurring between the first and third frames is used as an example intermediate frame that is compared to the predicted intermediate frame generated by the frame interpolation system 100.In one embodiment, the intermediate optical flow neural network model 122 and the flow refinement neural network models 145 are a fully convolutional neural network consisting of an encoder and a decoder with jumping connections between the encoder and decoder functions at the same spatial resolution. In one embodiment, the training loss function 210 evaluates one or more of a reconstruction loss, a perception loss, a distortion loss, and a smoothing loss.For given input frames (e.g., frames) I 0 and I 1, a set of intermediate frames between I 0 and I 1, where t i ∈(0,1), and the predicted intermediate frames, the loss function is a linear combination of four expressions:A reconstruction loss I r models how well the reconstruction of the intermediate individual images is:Such a reconstruction loss is defined in RGB space, in which the pixel values are in the range of [0, 255].Even if the L 1- loss is used to model the reconstruction error of intermediate frames, blurring may occur in the predictions. Thus, perceptual loss can be used to obtain details of the predictions and make interpolated images sharper. Specifically, the perception loss I p is defined asWhere φ denotes the conv4_3 features of a pre-trained VGG-16 (Visual Geometry Group) model.In addition to intermediate predictions, a distortion loss I w can also be included for modelling the quality of the calculated optical flow, defined asFinally, a smoothing term may be included to animate adjacent pixels to have similar flow values:The weights λ r, λ p, λ w and λ s may be empirically set as λ r= 0,8, λ p= 0,005, λ w= 0,4 and λ s= 1 using a validation set. In one embodiment, components of the frame interpolation systems 100, 150, and 200 are differentiable, including the flow calculation neural network model 210, the optical flow distortion units 105, the intermediate optical flow neural network models 142, and the flow refinement neural network models 102 and 122. Thus, the frame interpolation systems 100, 150, and 200 may be continuously trained.FIG. 2B illustrates a flow diagram of a method 220 for training the frame interpolation systems 100, 150 and 200 according to an embodiment. Although the method 220 is described in the context of a processing unit, the method 220 may also be performed by a program, a user-defined circuit, or by a combination of a user-defined circuit and a program. For example, the method 220 may be performed by a GPU (graphics processor unit), a CPU (central processor unit), or any processor capable of implementing a neural network model. Moreover, it will be understood by those skilled in the art that any system that performs method 220 is within the scope and scope of embodiments of the invention.Steps 162, 165, 170, and 135 are completed as described above in connection with FIG. 1F. In step 225, the predicted intermediate frame is compared to the baseline frame. In one embodiment, the training loss function 210 compares the intermediate frame to a baseline frame. In step 230, training may be completed based on differences between the predicted frame and the basic knowledge frame. A loss function, such as the loss function defined by equation (7), may be calculated to measure distances (i.e., differences or gradients) between the baseline frame and the predicted intermediate frame. In an embodiment, the loss function may be a combination of at least one of the reconstruction loss, the perception loss, the distortion loss, and the smoothing loss. The intermediate optical flow neural network model 142 and the flow interpolation neural network model 102 or 122 are considered sufficiently trained when the predicted frames generated for the input frames from the training dataset match the baseline frame or a threshold accuracy is achieved for the training dataset.When training is completed in step 230, frame interpolation systems 100, 150, and 200 may be employed to perform frame interpolation in step 240. Otherwise, if the training is not completed in step 230, then in step 235, the parameters are adjusted based on the loss function and steps 165 to 230 are repeated. In one embodiment, in step 235, the parameters of intermediate flow neural network model 142 and flow interpolation neural network model 102 or 122 are adjusted to reduce differences between the intermediate frame and the baseline frame. In one embodiment, the training loss function 210 adjusts the parameters of the intermediate flow neural network model 142 and the flow interpolation neural network model 102 or 122. By the method 220, the intermediate optical flow neural network model 142 and the flow interpolation neural network model 102 or 122 may be trained to learn the frame interpolation simultaneously.FIG. 2C illustrates input frames, bidirectional optical flows, and distorted input frames, according to an embodiment. In the input frames, the arms of the football player move down from T=0 to T=1. The area directly above the arm at T=0 is visible at t, but the same area is occluded (i.e., invisible) at T=1. The bidirectional optical flows are approximated based on the input frames and used to distort the input frames (shown in the last row).FIG. 2D illustrates visibility maps and predicted intermediate frames, according to an embodiment. Areas that are not visible are indicated by black in the visibility maps, and areas that are visible are indicated by white. The visibility maps in the top row clearly show that the area directly above the arm is visible at T=0 and occluded at T=1. The white area around the arms in V t←1 indicates pixels in I 1 that contribute most to the predicted I t while the corresponding occluded pixels (in black areas) in I 0 contribute little. Similar phenomena also occur around motion boundaries (e.g., around the body of the athletes).As discussed above, motion boundaries and strong occlusions are still a challenge for conventional frame interpolation methods. Therefore, the interpolated frames generated using conventional frame interpolation techniques tend to have artifacts around the boundaries of moving objects. In contrast, artifacts caused by motion boundaries and occlusions are reduced in intermediate frames predicted by the frame interpolation systems 100, 150, and 200. Conventional frame interpolation techniques rely on heuristic-based interflow calculation (i.e., flow interpolation) and occlusion augmentation, and are not continuously trainable. In contrast, the frame interpolation systems 100, 150 and 200 are continuously trainable. Further, intermediate frames may be predicted by the frame interpolation systems 100, 150, and 200 for simple video without requiring reference images or supervised training. Finally, the frame interpolation systems 100, 150 and 200 may be extended to predict multiple intermediate frames in parallel. In contrast, conventional frame interpolation techniques rely on recursion to calculate more than one intermediate frame and therefore can predict only one intermediate frame at a time.Parallel Processing ArchitectureFIG. 3 illustrates a parallel processing unit (PPU) 300 according to an embodiment. In one embodiment, the PPU 300 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 300 is a latency hiding architecture that has been developed to process many threads in parallel. A thread (i.e., a thread of execution) is an instantiation of a set of instructions configured to be executed by the PPU 300. In one embodiment, the PPU 300 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device, such as a liquid crystal display (LCD) device. In other embodiments, the PPU 300 may be used to perform general purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it is urgent to note that such a processor is provided for illustrative purposes only and that any processor may be used to supplement or substitute the same.One or more PPUs 300 may be configured to speed up thousands of high performance computing (HPC), data centers, and machine learning applications. The PPU 300 may be configured to accelerate numerous systems and applications for deep learning, including autonomous vehicle platforms, deep learning, high-precision voice, image, and text recognition systems, smart video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecast, big data analytics, astronomy, molecular dynamics simulation, financial modelling, robotics, factory automation, real-time voice translation, online search optimizations, and personalized user recommendations, and the like.As shown in FIG. 3, the PPU 300 includes an input / output (I / O) unit 305, a front end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more partitioning units 380. The PPU 300 may be connected to a host processor or other PPUs 300 via one or more high-speed NVLink 310 connections. The PPU 300 may be connected to a host processor or other peripheral devices via a link 302. The PPU 300 may also be connected to local memory that includes a number of memory devices 304. In one embodiment, the local memory may include a number of dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked within each device.The NVLink 310 connection allows systems to scale and include one or more PPUs 300 combined with one or more CPUs, supports cache coherency between the PPUs 300 and CPUs, and CPU mastering. Data and / or commands may be transmitted from the NVLink 310 via the hub 330 to / from other devices of the PPU 300, such as one or more copy engines, a video encoder, a video decoder, a power management device, etc. (not explicitly shown). The NVLink 310 will be described in more detail in connection with FIG. 5B.The I / O unit 305 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) over the link 302. The I / O unit 305 may communicate with the host processor directly via the connection 302 or via one or more intermediary devices, such as a memory bridge. In one embodiment, I / O unit 305 may communicate with one or more other processors, such as one or more of PPUs 300, via link 302. In one embodiment, I / O unit 305 implements a peripheral component interconnect express (PCIe) interface for communication over a PCIe bus, and link 302 is a PCIe bus. In alternative embodiments, I / O unit 305 may implement other types of well-known interfaces for communicating with external devices.The I / O unit 305 decodes packets received over the link 302. In one embodiment, the packets represent commands configured to cause the PPU 300 to perform various operations. The I / O unit 305 transmits the decoded instructions to various other units of the PPU 300 as indicated in the instructions. For example, some commands may be transmitted to the front end unit 315. Other commands may be transmitted to hub 330 or other devices of PPU 300, such as one or more copy engines, a video encoder, a video decoder, a power management device, etc. (not explicitly shown). In other words, the I / O unit 305 is configured to route communication between and among the various logical units of the PPU 300.In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides workloads to the PPU 300 for processing. A workload may include multiple instructions and data to be processed by those instructions. The buffer is a region in memory that is accessible from (i.e., reading / writing) both the host processor and the PPU 300. For example, I / O unit 305 may be configured to access the buffer in a system memory connected to link 302 via memory requests transmitted over link 302. In one embodiment, the host processor writes the command stream to the buffer and then transmits a pointer to the beginning of the command stream to the PPU 300. The front end unit 315 receives pointers to one or more command streams. The front end unit 315 manages one or more streams, reading commands from the streams and forwarding commands to the various units of the PPU 300.The front end unit 315 is coupled to a scheduler unit 320 that configures the various GPCs 350 to process tasks defined by one or more streams. The scheduler unit 320 is configured to track state information related to the various tasks managed by the scheduler unit 320. The state may indicate to which GPC 350 a task is associated, whether the task is active or inactive, a priority level associated with the task, and so forth. The scheduler unit 320 manages the execution of a plurality of tasks on the one or more GPCs 350.Scheduler unit 320 is coupled to a work distribution unit 325, which is configured to distribute tasks for execution on GPCs 350. The work distribution unit 325 may track a number of scheduled tasks received from the scheduling unit 320. In one embodiment, the work distribution unit 325 manages an open task pool and an active task pool for each of the GPCs 350. The open task pool may include a number of slots (e.g., 32 slots) that contain tasks to be processed by a particular GPC 350. The active task pool may include a number of slots (e.g., 4 slots) for tasks that are actively processed by the GPCs 350. When a GPC 350 completes execution of a task, that task is removed from the active task pool for the GPC 350 and one of the other tasks is selected from the open task pool and scheduled for execution on the GPC 350. If an active task on the GPC 350 was idle, e.g., while waiting for resolution of a data dependency, the active task may be removed from the GPC 350 and returned to the open task pool while another task in the open task pool is selected and scheduled for execution on the GPC 350.The work distribution unit 325 communicates with one or more GPCs 350 via the XBar 370. The XBar 370 is an interconnect network that couples many of the units of the PPU 300 to other units of the PPU 300. For example, the XBar 370 may be configured to couple the work distribution unit 325 to a particular GPC 350. Although not explicitly shown, one or more other units of the PPU 300 may also be connected to the XBar 370 via the hub 330.The tasks are managed by the scheduler unit 320 and sent from the work distribution unit 325 to a GPC 350. The GPC 350 is configured to process the task and generate results. The results may be taken over by other tasks within the GPC 350, passed over the XBar 370 to another GPC 350, or stored in the memory 304. The results may be written to memory 304 via partitioning units 380 that implement a memory interface for reading and writing data to / from memory 304. The results may be transmitted to another PPU 304 or CPU via the NVLink 310. In one embodiment, the PPU 300 includes a number U of partitioning units 380 equal to the number of separate and different memory devices 304 coupled to the PPU 300. A partitioning unit 380 will be described in more detail below in connection with FIG. 4B.In one embodiment, a host processor executes a driver core that implements an application programming interface (API) that enables one or more applications executing on the host processor to plan operations for execution on the PPU 300. In one embodiment, multiple computing applications are executed simultaneously by the PPU 300, and the PPU 300 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. An application may generate instructions (e.g., API calls) that cause the driver core to generate one or more tasks for execution by the PPU 300. The driver core outputs tasks to one or more streams that are processed by the PPU 300. Each task may include one or more groups of related threads, referred to herein as warp. In one embodiment, a warp 32 includes related threads that can be executed in parallel. Cooperating threads may refer to a plurality of threads, including instructions to perform the task, and may exchange data via shared memory. Threads and cooperating threads will be described in more detail in connection with FIG. 5A.FIG. 4A illustrates a GPC 350 of the PPU 300 of FIG. 3, according to an embodiment. As shown in FIG. 4A, each GPC 350 includes a series of hardware units for processing tasks. In one embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operations unit (PROP) 415, a raster engine 425, a work distribution crossbar (WDX) 480, a memory management unit (MMU) 490, and one or more data processing clusters (DPCs) 420. It should be appreciated that the GPC 350 of FIG. 4A may include additional hardware devices instead of or in addition to the devices shown in FIG. 4A.In one embodiment, the operation of GPC 350 is controlled by pipeline manager 410. The pipeline manager 410 manages the configuration of the one or more DPCs 420 for processing tasks assigned to the GPC 350. In one embodiment, the pipeline manager 410 may configure at least one of the one or more DPCs 420 to implement at least a portion of a graphics rendering pipeline. For example, a DPC 420 may be configured to execute a vertex shader program on the programmable streaming multiprocessor (SM) 440. The pipeline manager 410 may also be configured to route packets received from the work distribution unit 325 to the appropriate logical units within the GPC 350. For example, some packets may be forwarded to fixed function hardware units in the PROP 415 and / or raster engine 425, while other packets may be forwarded to the DPCs 420 for processing by the primitive engine 435 or the SM 440. In an embodiment, the pipeline manager 410 may configure at least one of the one or more DPCs 420 to implement a neural network model and / or a computational pipeline.The PROP unit 415 is configured to route the data generated by the raster engine 425 and the DPCs 420 to a raster operations (ROP) unit, which will be described in more detail in connection with FIG. 4B. The PROP unit 415 may also be configured to perform optimizations for color mixing, organize pixel data, perform address translations, and the like.The raster engine 425 includes a series of fixed function hardware units configured to perform various raster operations. In one embodiment, raster engine 425 includes a setup engine, a coarse raster engine, a pop engine, a clipping engine, a fine raster engine, and a tile merging engine. The setup engine receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are transmitted to the coarse raster engine to generate coverage information (e.g., an x, y coverage mask for a tile) for the primitive. The output of the coarse raster engine is transmitted to the extraction engine, where fragments associated with the primitive that fail a z-test are extracted and sorted out, respectively, and to a clipping engine, where fragments that are outside of a viewing cone are clipped. The fragments that survive clipping and discarding may be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of raster engine 425 includes fragments to be processed by, for example, a fragment shader implemented in a DPC 420.Each DPC 420 included in the GPC 350 includes an M-pipe controller (MPC) 430, a root function engine 435, and one or more SMs 440. The MPC 430 controls the operation of the DPC 420 and forwards the packets received from the pipeline manager 410 to the appropriate units of the DPC 420. For example, packets associated with a vertex may be forwarded to the primitive engine 435, which is configured to fetch vertex attributes associated with the vertex from the memory 304. In contrast, packets associated with a shader program may be transmitted to the SM 440.The SM 440 includes a programmable streaming processor configured to process tasks represented by a series of threads. Each SM 440 is multi-threaded and configured to execute a plurality of threads (e.g., 32 threads) from a particular group of threads simultaneously. In one embodiment, the SM 440 implements a single instruction, multiple data (SIMD) architecture in which each thread in a group of threads (i.e., a warp) is configured to process a different dataset based on the same instruction set. All threads in the group of threads execute the same instructions. In another embodiment, the SM 440 implements a single instruction, multiple thread (SIMT) architecture in which each thread in a group of threads is configured to process a different dataset based on the same instruction set, but individual threads in the group of threads may diverge during execution. In one embodiment, a program counter, a fetch stack, and an execution state are maintained for each warp, thereby allowing simultaneity between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, a fetch stack, and an execution state are maintained for each individual thread, thereby allowing equal simultaneity between all threads, within and between warps. If the execution state is maintained for each individual thread, threads executing the same instructions may be converged and executed in parallel to achieve maximum efficiency. The SM 440 will be described in more detail below in connection with FIG. 5A.The MMU 490 provides an interface between the GPC 350 and the partitioning unit 380. The MMU 490 may provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 490 provides one or more translation lookaside buffers (TLBs) to perform a translation from virtual addresses to physical addresses in the memory 304.FIG. 4B illustrates a memory partitioning unit 380 of the PPU 300 of FIG. 3, according to an embodiment. As shown in FIG. 4B, the memory partitioning unit 380 includes a raster operations (ROP) unit 450, a level 2 (L2) cache 460, and a memory interface 470. The memory interface 470 is coupled to the memory 304. The memory interface 470 may implement 32, 64, 128, 1024-bit data buses or the like for high-speed data transfer. In one embodiment, the PPU 300 includes U-memory interfaces 470, one memory interface 470 per pair of partitioning units 380, each pair of partitioning units 380 connected to a corresponding memory device 304. For example, the PPU 300 may be connected to up to Y memory devices 304, such as high bandwidth memory stacks or double data rate graphics cards, version 5, synchronous dynamic random access memories, or other types of persistent memory.In one embodiment, the memory interface 470 implements an HBM2 memory interface and Y is equal to half of U. In one embodiment, the HBM2 memory stacks are located in the same physical package as the PPU 300, thereby providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory chips and Y is 4, where the HBM2 stack includes two 128-bit channels per chip for a total of 8 channels and a 1024-bit data bus width.In one embodiment, memory 304 supports single-error correcting double-error detecting (SECDED) error correction code (ECC) to protect the data. ECC provides higher reliability for computing applications that are sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPUs 300 process very large sets of data and / or operate applications over extended periods of time.In one embodiment, the PPU 300 implements a multi-level memory hierarchy. In one embodiment, memory partition unit 380 supports unified memory to provide a single unified virtual address space for CPU and PPU 300 memories that facilitates data exchange between virtual memory systems. In one embodiment, the frequency of accesses of a PPU 300 to memory residing on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 300 that more frequently accesses the pages. In one embodiment, NVLink 310 supports address translation services that allow PPU 300 to directly access a CPU's page tables and provide full access to CPU memory through PPU 300.In one embodiment, copy engines transfer data between multiple PPUs 300 or between PPUs 300 and CPUs. The copy engines may generate page errors for addresses not mapped in the page tables. The memory partition unit 380 may then remedy the page errors, mapping the addresses into the page table, whereupon the copy engine may perform the transfer. In a conventional system, memory for multiple copy engine operations is anchored (i.e., not removable) between multiple processors, which significantly reduces available memory. With hardware page fault discard, addresses can be passed to the copy engines without concern of whether the memory pages are resident and the copy operation is transparent.Data from memory 304 or other system memory may be retrieved from memory partitioning unit 380 and stored in the on-chip L2 cache 460 shared by the various GPCs 350. As shown, each memory partitioning unit 380 includes a portion of the L2 cache 460 associated with a corresponding memory device 304. Subordinate caches may then be implemented in different units within the GPCs 350. For example, each of the SMs 440 may implement a level one (L1) cache. The L1 cache is a private memory dedicated to a particular SM 440. Data from the L2 cache 460 may be fetched and stored in each of the L1 caches for processing in the functional units of the SM 440. L2 cache 460 is coupled to memory interface 470 and XBar 370.The ROP unit 450 performs graphics raster operations with respect to pixel colors, such as color compression, pixel blending, and the like. The ROP unit 450 also implements depth tests associated with the raster engine 425, receiving a depth for a sample location associated with a pixel fragment from the sample engine of the raster engine 425. The depth is tested against a corresponding depth in a depth buffer for a sample location associated with the fragment. If the fragment passes the depth test for the sample location, the ROP unit 450 updates the depth buffer and transmits a result of the depth test to the raster engine 425. Note that the number of partitioning units 380 may be different than the number of GPCs 350, and thus each ROP unit 450 may be coupled to each of the GPCs 350. The ROP unit 450 tracks the packets received from the various GPCs 350 and determines the GPC 350 to which a result generated by the ROP unit 450 is routed via the Xbar 370. Although the ROP unit 450 is included within the memory partitioning unit 380 in FIG. 4B, in another embodiment, the ROP unit 450 may be external to the memory partitioning unit 380. For example, the ROP unit 450 may reside in the GPC 350 or in another unit.FIG. 5A illustrates the streaming multiprocessor 440 of FIG. 4A, according to an embodiment. As shown in FIG. 5A, the SM 440 includes an instruction cache 505, one or more scheduler units 510, a register file 520, one or more processing cores 550, one or more special function units (SFUs) 552, one or more load / store units (LSUs) 554, an interconnect network 580, and a shared memory / L1 cache 570.As described above, work distribution unit 325 sends tasks for execution on GPCs 350 of PPU 300. The tasks are associated with a particular DPC 420 within a GPC 350, and if the task is associated with a shader program, the task may be associated with an SM 440. Scheduler unit 510 receives the tasks from work distribution unit 325 and manages instruction scheduling for one or more of the thread blocks associated with SM 440. Scheduler unit 510 schedules thread blocks for execution as warps of parallel threads, with at least one warp associated with each thread block. In one embodiment, each warp executes 32 threads. Scheduler unit 510 may manage a plurality of different thread blocks, allocating the warps to the different thread blocks, and then sending instructions from the plurality of different cooperative groups to the different functional units (i.e., cores 550, SFUs 552, and LSUs 554) during each clock cycle.Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, thereby enabling the expression of richer, more efficient parallel decompositions. Cooperative boot APIs support synchronization between thread blocks for execution of parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (i.e., the syncthreads() function). However, programmers often wish to define groups of threads that are less than the granularity of threads and synchronize within the defined groups to enable more performance, design flexibility, and software re-useability in the form of common group-wide function interfaces.Cooperative Groups allow programmers to explicitly define groups of threads at sub-block (i.e., as small as a single thread) and multi-block granularities and perform collective operations such as synchronization on the threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local context without having to make assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire network of thread blocks.A transmitting unit 515 is configured to transmit instructions to one or more of the functional units. In the embodiment, scheduler unit 510 includes two transmit units 515 that allow two different instructions of the same warp to be transmitted during each clock cycle. In alternative embodiments, each scheduler unit 510 may include a single transmit unit 515 or additional transmit units 515.Each SM 440 includes a register file 520 that provides a set of registers for the functional units of the SM 440. In one embodiment, the register file 520 is partitioned between each of the functional units such that each functional unit is associated with a dedicated portion of the register file 520. In another embodiment, the register file 520 is partitioned between the various warps being executed by the SM 440. Register file 520 provides a cache for operands connected to the functional unit data paths.Each SM 440 includes L processing cores 550. In one embodiment, the SM 440 includes a large number (e.g., 128, etc.) of different processing cores 550. Each core 550 may include a fully pipetted single precision, double precision, and / or mixed precision processing unit that includes a floating point computational logic unit and an integer computational logic unit. In one embodiment, the floating point computational logic units implement the IEEE 754-2008 standard for floating point arithmetic. In one embodiment, cores 550 include 64 single precision (32-bit) floating point cores, 64 integer cores, 32 double precision (64-bit) floating point cores, and 8 tensor cores.Tensor cores configured to perform matrix operations and, in one embodiment, one or more tensor cores are included in cores 550. In particular, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing. In one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4x4 matrices.In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiplication requires 64 operations and results in a high precision product which is then accumulated using the 32-bit floating point addition with the other intermediates for 4x4x4x4 matrix multiplication. In practice, tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations that are composed of these smaller elements. An API, such as the CUDA 9 C++ API, provides special matrix loads, matrix multiplications and accumulations, and matrix storage operations to efficiently utilize tensor cores from a CUDA C++ program. At the CUDA level, the warp level interface starts from 16x16 size matrices spanning all 32 threads of the warp.Each SM 440 also includes M SFUs 552 that perform specific functions (e.g., attribute scoring, reciprocal square root, and the like). In an embodiment, the SFUs 552 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 552 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture units are configured to load texture maps (e.g., a 2D array of texels) from memory 304 and sample the texture maps to generate sampled texture values for use in shader programs executed by SM 440. In one embodiment, the texture maps are stored in the shared memory / L1 cache 470. The texture units implement texture operations such as filtering operations using mip-maps (i.e., texture maps with varying level of detail). In one embodiment, each SM 340 includes two texture units.Each SM 440 also includes N LSUs 554 that implement load and store operations between the shared memory / L1 cache 570 and the register file 520. Each SM 440 includes an interconnect network 580 that connects each of the functional units to the register file 520 and the LSU 554 to the register file 520 and the shared memory / L1 cache 570. In one embodiment, interconnect network 580 is a crossbar that can be configured to connect any of the functional units to any of the registers in register file 520 and connect LSUs 554 to the register file and memory locations in shared memory / L1 cache 570.Shared memory / L1 cache 570 is an array of on-chip memory memories that enables data storage and communication between SM 440 and primitive engine 435, as well as between threads in SM 440. In one embodiment, shared memory / L1 cache 570 includes 128MB memory capacity and resides in the path from SM 440 to partition unit 380. Shared memory / L1 cache 570 may be used to cache reads and writes. One or more of shared memory / L1 cache 570, L2 cache 460, and memory 304 are backup memories.Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory accesses. The capacity is usable as a cache for programs that do not use shared memory. For example, if shared memory is configured to use half the capacity, texture and load / store operations may use the remaining capacity. Integration within shared memory / L1 cache 570 allows shared memory / L1 cache 570 to operate as a high-throughput line for streaming data while simultaneously allowing high-bandwidth, low-latency access to frequently reused data.In a configuration for universal parallel computations, a simpler configuration compared with graphics processing may be used. In particular, the fixed function graphics processing units shown in FIG. 3 are bypassed, creating a much simpler programming model. In the universal parallel computation configuration, the work distribution unit 325 allocates and distributes thread blocks directly to the DPCs 420. The threads in a block execute the same program, wherein a unique thread ID is used in the computation to ensure that each thread generates unique results, and wherein the SM 440 is used to execute the program and perform computations, the shared memory / L1 cache 570 is used to communicate between threads, and the LSU 554 is used to read and write global memory via the shared memory / L1 cache 570 and the memory partition unit 380. In the configuration for universal parallel computations, the SM 440 may also write commands that the scheduler unit 320 may use to begin new work on the DPCs 420.The PPU 300 may be incorporated into a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g., a wireless, portable device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-screen, a portable electronic device, and the like. In an embodiment, the PPU 300 is embodied on a single semiconductor substrate. In another embodiment, the PPU 300 is included in a system on a chip (SoC) along with one or more other devices, such as additional PPUs 300, memory 204, a reduced instruction set CPU (RISC), a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.In one embodiment, the PPU 300 may be included on a graphics card that includes one or more memory devices 304. The graphics card may be configured to connect to a PCIe slot on a motherboard of a desktop computer. In yet another embodiment, the PPU 300 may be an integrated graphics processing unit (iPU) or a parallel processor included in the chipset of the motherboard.Example Computing SystemSystems with multiple GPUs and CPUs are used in a variety of industries as developers provide and utilize more parallelism in applications such as artificial intelligence. Powerful GPU-accelerated systems with tens of thousands and more compute nodes are deployed in data centers, research facilities, and supercomputers to solve ever-increasing problems. As the number of processing devices within the high performance systems increases, the communication and data transmission mechanisms must be scaled to support the increased bandwidth.FIG. 5B is a conceptual diagram of a processing system 500 implemented using the PPU 300 of FIG. 3, according to one embodiment. The example system 565 may be configured to implement the methods 100, 160, and 220 illustrated in FIGS. 1C, 1F, and 2B. The processing system 500 includes a CPU 530, a switch 510, and a plurality of PPUs 300 and corresponding memories 304, respectively. The NVLink 310 provides high speed communication links between each of the PPUs 300. Although a certain number of NVLink 310 and interconnect 302 connections are illustrated in FIG. 5B, the number of connections to each PPU 300 and CPU 530 may vary. Switch 510 connects between link 302 and CPU 530. The PPUs 300, the memories 304, and the NVLinks 310 may reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to connect between different different connections and / or links.In another embodiment (not shown), the NVLink 310 provides one or more high speed communication links between each of the PPUs 300 and the CPU 530, and connects the switch 510 between the link 302 and each of the PPUs 300. The PPUs 300, the memories 304, and the interconnect 302 may be disposed on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), the link 302 provides one or more communication links between each of the PPUs 300 and the CPU 530, and connects the switch 510 between each of the PPUs 300 using the NVLink 310 to provide one or more high-speed communication links between the PPUs 300. In another embodiment (not shown), the NVLink 310 provides one or more high speed communication links between the PPUs 300 and the CPU 530 via the switch 510. In yet another embodiment (not shown), the link 302 directly provides one or more communication links between each of the PPUs 300. One or more of the high speed communication links over the NVLink(s) 310 may be implemented as a physical NVLink link or as either an on-chip or an on-die link using the same protocol as the NVLink 310.In the context of the present description, a single semiconductor platform may refer to a single unitary semiconductor-based integrated circuit fabricated on a die or chip. It is noted that the term single semiconductor platform may also refer to multichip modules with increased connectivity that simulate on-chip operation and provide substantial improvements over the use of a conventional bus implementation. Of course, the various circuits or devices may also be arranged individually or in various combinations of semiconductor platforms as desired by the user. Alternatively, the parallel processing module 525 may be implemented as a printed circuit board substrate, and each of the PPUs 300 and / or each of the memories 304 may be a packaged device. In one embodiment, the CPU 530, the switch 510, and the parallel processing module 525 are located on a single semiconductor platform.In one embodiment, the signal transmission rate of each NVLink 310 is 20 to 25 gigabits / second, and each PPU 300 includes six NVLink 310 interfaces (as shown in FIG. 5B, five NVLink 310 interfaces are included for each PPU 300). Each NVLink 310 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 300 gigabytes / second. The NVLinks 310 may be used exclusively for PPU-to-PPU communication, as shown in FIG. 5B, or for a combination of PPU-to-PPU and PPU-to-CPU, if the CPU 530 also includes one or more NVLink 310 interfaces.In one embodiment, NVLink 310 enables direct load / store / core access from CPU 530 to memory 304 of each PPU 300. In one embodiment, NVLink 310 supports coherency operations so that data read from memories 304 may be stored in the cache hierarchy of CPU 530, thereby reducing cache access latency for CPU 530. In one embodiment, NVLink 310 includes Address Translation Services (ATS) support so that PPU 300 can directly access page tables within CPU 530. One or more of the NVLinks 310 may also be configured for operation in a low-consumption mode.FIG. 5C illustrates an example system 565 in which the various architecture and / or functionality of the various previous embodiments may be implemented. The example system 565 may be configured to implement the methods 100, 160, and 220 illustrated in FIGS. 1C, 1F, and 2B.As shown, a system 565 is provided that includes at least one central processor unit 530 connected to a communication bus 575. Communication bus 575 may be implemented using any suitable protocol, such as peripheral component interconnect (PCI), PCI express, accelerated graphics port (AGP), HyperTransport, or any other bus or point-to-point communication protocol(s). The system 565 further includes a main memory 540. Control logic (software) and the data are stored in main memory 540, which may be embodied as random access memory (RAM).The system 565 further includes input devices 560, the parallel processing system 525, and display devices 545, i.e., a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, or the like. User input may be received from the input devices 560, e.g., keyboard, mouse, touchpad, microphone, and the like. Each of the foregoing modules and / or devices may even be located on a single semiconductor platform to form the system 565. Alternatively, the various modules may be arranged individually or in various combinations of semiconductor platforms as desired by the user.Further, the system 565 may be communicatively coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN), such as the Internet, a peer-to-peer network, a cable network, or the like) via a network interface 535.The system 565 may further include a secondary storage (not shown). The secondary storage 610 includes, for example, a hard disk drive and / or a removable storage drive that is a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads and / or writes from / to a removable storage unit in a known manner.Computer programs, or computer control logic algorithms, may be stored in main memory 540 and / or secondary storage. When executed, such computer programs allow the system 565 to perform various functions. The memory 540, the memory, and / or any other memory are possible examples of computer readable media.The architecture and / or functionality of the various previous figures may be implemented in the context of a universal computer system, a board system, a game console system dedicated for entertainment purposes, an application specific system, and / or any other desired system. The system 565 may take the form of, for example, a desktop computer, a laptop computer, a tablet computer, servers, super computers, a smart phone (e.g., a wireless, portable device), a personal digital assistant (PDA), a digital camera, a vehicle, a head mounted display, a portable electronic device, a mobile telephone device, a television, a workstation, game consoles, an embedded system, and / or any other type of logic.While various embodiments have been described above, it is to be understood that these have been presented by way of example only and not as limiting. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.Machine LearningDeep neural networks (DNNs) developed on processors such as the PPU 300 have been used for various applications, from self-driving cars to faster drug development, from automatic imaging in online image databases to intelligent real-time voice translation in video chat applications. Deep learning is a technique that models the human brain neural learning process, continuously learns, continuously smarts, and over time provides faster and more accurate results. A child is first taught by an adult to properly identify and classify various shapes to ultimately identify shapes without any follow-up. Likewise, a deep learning or neural learning system must be trained in object recognition and classification to allow it to identify more intelligent and efficient basic objects, occluded objects, etc., and simultaneously assign context to the objects.At the simplest level, neurons in the human brain view various inputs that are received, are assigned levels of importance to each of these inputs, and one output is passed to other neurons to respond thereto. An artificial neuron or perceptron is the most basic model of a neural network. In one example, a perceptron may receive one or more inputs representing various features of an object for which the perceptron is trained to recognize and classify them, and each of these features is assigned a particular weight based on the meaning of that feature for defining the shape of an object.A deep neural network (DNN) model includes multiple layers of many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained with enormous amounts of input data to quickly solve complex problems with high accuracy. In an example, a first layer of the DNN model decomposes an input image of a car into different sections and searches for basic patterns such as lines and angles. The second layer assembles the lines to search for higher level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate a label for the input image that identifies the model of a particular automobile make.Once the DNN is trained, the DNN can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of conclusions (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks that are deposited in automated teller machines, identifying images of friends on photos, providing movie recommendations to over fifty million users, identifying and classifying various types of automobiles, pedestrians, and road hazards in automated guided cars, or translating human speech in real-time.During training, data flows through the DNN in a forward propagation phase until a prediction is made that indicates a label corresponding to the input. If the neural network does not correctly identify the input, errors between the correct identification and the predicted identification are analyzed and the weights for each feature are adjusted during a backpropagation phase until the DNN correctly identifies the input and other inputs in a training dataset. Training complex neural networks requires enormous amounts of parallel computing power, including floating point multiplications and additions supported by the PPU 300. Inferencing is less computationally intensive than training because it is a latency sensitive process in which a trained neural network is applied to new inputs that have not been so far to classify images, translate speech, and generally derive new information.Neural networks rely heavily on matrix math operations, and complex layered networks require enormous amounts of floating point performance and bandwidth for both efficiency and speed. With thousands of compute cores optimized for matrix math operations and delivering hundreds of TFLOPS of power, the PPU 300 is a computing platform capable of delivering the power required for deep neural network-based artificial intelligence and machine learning applications.

Claims

A computer-implemented method comprising: receiving forward optical flow data (F 0→1), calculated from a first frame (I 0) toward a second frame (I 1) in a sequence of frames, and forward intermediate optical flow data (F̂ t→0) for a time in the sequence of frames that is between the first frame and the second frame; receiving backward optical flow data calculated from the second frame toward the first frame and backward intermediate optical flow data (F̂ t→1) for the time; and generating, by a neural network model for flow interpolation (102, 122), an intermediate frame (I t) at the time based on the first frame (I 0) and the second frame (I 1), the forward intermediate optical flow data (F̂ t→0), the backward intermediate optical flow data (F̂ t→1), the first frame distorted in accordance with the forward intermediate optical flow data (F̂ t→0) (Lithographically 0→t), and the second frame distorted in accordance with the backward intermediate optical flow data (F̂ t→1) (Lithographically 1→t), further comprising interpolating, by the flow interpolating neural network model (102, 122), between the forward optical flow data (F 0→1) and the backward optical flow data to generate the forward intermediate optical flow data (F̂ t→o) and the backward intermediate optical flow data (F̂ t→1) at the time.The computer-implemented method of claim 1, further comprising generating, by a neural network model for flow calculation (215), the forward optical flow data (F 0→1) and the backward optical flow data based on the first frame (I 0) and the second frame (I 1).The computer-implemented method of claim 2, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102, 122) are trained jointly unsupervised.The computer-implemented method of claim 2, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102, 122) are jointly trained using a loss function that minimizes perceptual loss.The computer-implemented method of claim 2, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102, 122) are jointly trained using a loss function that minimizes reconstruction loss.The computer-implemented method of claim 2, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102, 122) are jointly trained using a loss function that minimizes distortion loss.The computer-implemented method of claim 2, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102, 122) are trained together using a loss function that minimizes a smoothing loss.The computer-implemented method of any of claims 1 to 7, wherein the neural network model for flow calculation (215) and the neural network for flow interpolation (102, 122) are convolutional neural network models.The computer-implemented method of any of claims 1-8, wherein the flow interpolation neural network model (102, 122) is based on the forward intermediate optical flow data (F̂ t→o), the backward intermediate optical flow data (F̂ t→1), the first input frame (I 0), the second input frame (I 1), the first frame distorted (Lithographically 0→t), in accordance with the forward intermediate optical flow data (F̂ t→0) and the second frame, which has been distorted in accordance with the backward optical interflow data (F̂ t→1) (Diagrammatically 1→t), a forward visibility map and a backward visibility map are calculated.The computer-implemented method of any of claims 1-9, further comprising: receiving second forward optical interflow data (F̂ t→0) for a second time in the sequence of frames that is between the first frame (I 0) and the second frame (I 1) ; receiving second backward optical interflow data (F̂ t→1) for the second time; generating, by the flow interpolating neural network model (102, 122), a second intermediate frame (I t) at the second time based on the second forward intermediate optical flow data (F̂ t→o), the second backward intermediate optical flow data (F̂ t→1), the first frame distorted in accordance with the second forward intermediate optical flow data (F̂ t→0) (Lithographically 0→t), and the second frame distorted in accordance with the second backward intermediate optical flow data (F̂ t→1) (Lithographically 1→t).The computer-implemented method of any of claims 1-10, wherein the flow interpolation neural network model (102, 122) generates the intermediate frame (I t) at the time by linearly merging the first frame distorted (Lithographically 0→t), in accordance with the forward intermediate optical flow data (F̂ t→0) and the second frame distorted (Lithographically 1→t\), in accordance with the backward intermediate optical flow data (F̂ t→1).The computer-implemented method of claim 11, further comprising, prior to the linear fusing, applying a forward visibility map to the first frame distorted in accordance with the forward intermediate optical flow data (F̂ t→0) (Lithographically 0→t), and applying a rearward visibility map to the second frame distorted in accordance with the rearward intermediate optical flow data (F̂ t→1) (Lithographically 1→t).A system comprising: a flow interpolation neural network model (102, 122) configured to: receive forward optical flow data (F 0→1), calculated from a first frame (I 0) to a second frame (I 1) in a sequence of frames, and forward intermediate optical flow data (F̂ t→0) for a time in the sequence of frames that is between the first frame and the second frame; receiving backward optical flow data calculated from the second frame (I 1) toward the first frame (I 0) and backward intermediate optical flow data (F̂ t→1) for the time; and generating an intermediate frame (I t) at the time based on the first frame and the second frame, the forward intermediate optical flow data (F̂ t→o), the backward intermediate optical flow data (F̂ t→1), of the first frame distorted in accordance with the forward intermediate optical flow data (F̂ t→0) (Lithographically 0→t), and the second frame distorted (Lithographically 1→t), in accordance with the backward optical interflow data (F̂ t→1) further comprising an intermediate flow neural network model (142) configured to interpolate between the forward optical flow data (F 0→1) and the backward optical flow data to generate the forward optical interflow data (F̂ t→0) and the backward optical interflow data (F̂ t→1) at the time.The system of claim 13, further comprising a flow calculation neural network model (215) configured to generate the forward optical flow data (F 0→1) and the backward optical flow data based on the first frame (I 0) and the second frame (I 1).The system of claim 14, wherein the flow calculation neural network model (215) and the flow interpolation neural network model (102 or 122) are trained jointly unsupervised.The system of any of claims 13 to 15, wherein the flow interpolation neural network model (102, 122) calculates a forward visibility map and a rearward visibility map based on the forward intermediate optical flow data (F̂ t→0), the rearward intermediate optical flow data (F̂ t→1), the first input frame, the second input frame, the first frame distorted (Lithographically 0→t), in accordance with the forward intermediate optical flow data (F̂ t→0) and the second frame distorted (Lithographically 1→t), in accordance with the rearward intermediate optical flow data (F̂ t→1).The system according to any one of claims 13 to 16, wherein the flow interpolation neural network model (102, 122) generates the intermediate frame (I t) at the time by linearly merging the first frame distorted (Lithographically 0→t), in accordance with the forward intermediate optical flow data (F̂ t→0) and the second frame distorted (Lithographically 1→t\), in accordance with the backward intermediate optical flow data (F̂ t→1).A non-transitory computer readable storage medium storing instructions which, when executed by a processing unit, cause the processing unit to perform a computer implemented method according to any one of claims 1 to 12.

Images