Video Frame Synthesis Using Tensor Neural Networks

Abstract

A method for achieving video frame synthesis using a tensor neural network includes receiving input video data including one or more missing frames, transforming the input video data into an input tensor, and generating output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including multiple phases that realizes an iterative tensor shrinkage-threshold algorithm (ISTA) through tensor completion based on the input tensor, and obtaining a loss function based on the output video data.

Description

[Technical Field]

[0001] The present invention relates generally to artificial intelligence, and more specifically to video frame synthesis using tensor neural networks. [Background technology]

[0002] Video frame synthesis is a computer vision task that has attracted widespread interest. During video frame synthesis, new video frames are constructed from existing video. Video frame synthesis can be divided into two categories: video frame interpolation and video frame prediction. Video frame interpolation constructs new video frames by restoring missing frames between existing video frames, while video frame prediction constructs new video frames by generating future frames from previous frames. Video frame synthesis techniques have a variety of applications. For example, video frame synthesis techniques can be used to convert standard video into high-quality slow-motion video with higher frame rates and smoother view transitions, to compensate for distortion caused by camera vibration during recording using video stabilization, and to mitigate the problem of missing video frames in wireless video transmission using motion smoothing (e.g., in unmanned aerial vehicle (UAV) and virtual reality (VR) applications).

[0003] Existing video frame synthesis solutions suffer from one or more drawbacks. For example, existing neural network-based methods do not clearly provide low-rank tensors for capturing spatiotemporal correlations of video frames in high-dimensional space, potentially lacking a mathematical interpretation. In addition, compressed sensing-based iterative algorithms typically require manual parameter generation and relatively long execution times for high-quality video frame synthesis. For example, some existing optimization algorithms can require over an hour to process a one-minute video (30 frames per second) with 100 missing frames. Therefore, it would be advantageous to provide a video synthesis solution that addresses at least the above issues. Summary of the Invention

[0004] A method for achieving video frame synthesis using a tensor neural network includes receiving input video data including one or more missing frames, converting the input video data into an input tensor, and generating output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including multiple phases that realizes an iterative shrinkage thresholding algorithm (ISTA) through tensor completion based on the input tensor, and obtaining a loss function based on the output video data.

[0005] In an embodiment of this method, generating an output tensor includes, for a given one of the phases of TTNet, updating intermediate synthesis results in the original domain, transforming the intermediate synthesis results in the original domain into transformed intermediate synthesis results in the transformed domain, applying a soft threshold based on the transformed intermediate synthesis results to generate synthesized video data in the transformed domain, and transforming the synthesized video data in the transformed domain back to the original domain using an inverse transform.

[0006] In an embodiment of this method, an intermediate synthesis result in the original domain is defined based in part on the image tensors and observation tensors received by a given one of the phases.

[0007] In an embodiment of this method, transforming the intermediate synthesis result in the original domain into a transformed intermediate synthesis result includes applying a first convolution to the intermediate synthesis result, applying an activation function to an output of the first convolution, and applying a second convolution to an output of the activation function to generate the transform. In one embodiment, the first and second convolutions include two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU).

[0008] In an embodiment of this method, applying a soft threshold based on the transformed intermediate composite result to generate composite image data in the transform domain includes applying multiple soft threshold operations in parallel to each frontal slice of the transformed intermediate composite result, and overlapping outputs of the multiple soft threshold operations to generate composite image data in the transform domain.

[0009] In an embodiment of this method, transforming the synthetic video data in the transform domain back to the original domain using an inverse transform includes applying a third convolution to the synthetic video data in the transform domain, applying a second activation function to an output of the third convolution, and applying a fourth convolution to an output of the second activation function. In one embodiment, the third and fourth convolutions include 2D multi-channel convolutions with different kernels, and the activation function is a ReLU.

[0010] In an embodiment of this method, transforming the synthetic video data in the transform domain back to the original domain using an inverse transform includes applying a first convolution to the synthetic video data in the transform domain, applying an activation function to an output of the first convolution, and applying a second convolution to an output of the activation function. In one embodiment, the first and second convolutions include 2D multi-channel convolutions with different kernels, and the activation function is a ReLU.

[0011] In an embodiment of this method, the loss function includes a first part that evaluates the accuracy of one or more synthetic frames and a second part that accounts for sparsity. More specifically, the loss function may be a linear combination of the first and second parts.

[0012] A system for implementing video synthesis using a tensor neural network includes a memory configured to store program code and at least one processor device operatively coupled to the memory, wherein the at least one processor device is configured to execute the program code stored in the memory device to: receive input video data including one or more missing frames; transform the input video data into an input tensor; generate output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including multiple phases that implements a tensor iterative shrinkage-threshold algorithm (ISTA) through tensor completion based on the input tensor; and obtain a loss function based on the output video data.

[0013] In an embodiment of this system, at least one processor device is configured to generate an output tensor for a given one of a plurality of phases of TTNet by updating intermediate synthesis results in the original domain, transforming the intermediate synthesis results in the original domain into transformed intermediate synthesis results in the transformed domain, applying a soft threshold based on the transformed intermediate synthesis results to generate synthesized video data in the transformed domain, and transforming the synthesized video data in the transformed domain back to the original domain using an inverse transform.

[0014] In an embodiment of this system, an intermediate synthesis result in the original domain is defined based in part on the image tensors and observation tensors received by a given one of the phases.

[0015] In an embodiment of this system, the at least one processor device is configured to transform the intermediate synthesis result in the original domain into a transformed intermediate synthesis result by applying a first convolution to the intermediate synthesis result, applying an activation function to an output of the first convolution, and applying a second convolution to an output of the activation function to generate a transform. In one embodiment, the first and second convolutions include two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU).

[0016] In an embodiment of this system, at least one processor device is configured to apply soft thresholds based on the transformed intermediate composite result to generate composite video data in the transform domain by applying multiple soft threshold operations in parallel to each frontal slice of the transformed intermediate composite result and overlapping outputs of the multiple soft threshold operations to generate composite video data in the transform domain.

[0017] In an embodiment of this system, the at least one processor device is configured to transform the synthetic video data in the transform domain back to the original domain using an inverse transform by applying a first convolution to the synthetic video data in the transform domain, applying an activation function to an output of the first convolution, and applying a second convolution to an output of the activation function. In one embodiment, the first and second convolutions include 2D multi-channel convolutions with different kernels, and the activation function is a ReLU.

[0018] In an embodiment of this system, the loss function includes a first part that evaluates the accuracy of one or more synthetic frames and a second part that accounts for sparsity. More specifically, the loss function may be a linear combination of the first and second parts.

[0019] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0020] The following description provides details of preferred embodiments with reference to the following drawings: [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block / flow diagram illustrating a high-level overview of a video frame synthesis system using a transform-based tensor neural network (TTNet), according to an embodiment of the present invention. [Figure 2] 2 is a block / flow diagram illustrating modules of the TTNet phases of FIG. 1 in accordance with an embodiment of the present invention. [Figure 3] 3 is a block / flow diagram illustrating an exemplary transformation module of the phase of FIG. 2 in accordance with an embodiment of the present invention. [Figure 4] 3 is a block / flow diagram illustrating an exemplary soft threshold module of the phase of FIG. 2 in accordance with an embodiment of the present invention. [Figure 5] 3 is a block / flow diagram illustrating an exemplary inverse transform module of the phase of FIG. 2 in accordance with an embodiment of the present invention. [Figure 6] FIG. 1 illustrates exemplary pseudocode illustrating an Iterative Tensor Shrinkage Threshold Algorithm (ISTA), in accordance with an embodiment of the present invention. [Figure 7] 1 is a block / flow diagram illustrating a system / method for implementing video frame synthesis using a transform-based tensor neural network (TTNet) implementing the tensor iterative shrinkage-threshold algorithm (ISTA), according to an embodiment of the present invention. [Figure 8] 8 is a block / flow diagram illustrating a system / method for implementing the tensor ISTA inference iteration of FIG. 7 according to an embodiment of the present invention. [Figure 9] 1 is a block diagram illustrating a processing system according to an embodiment of the present invention. [Figure 10]FIG. 1 is a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to one embodiment. [Figure 11] FIG. 1 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Embodiments of the present invention use tensor neural networks to perform video synthesis. More specifically, the embodiments described herein can construct a transform-based tensor neural network (TTNet) by representing video data as tensors and deploying the Iterative Tensor Shrinkage and Thresholding Algorithm (ISTA) into a multiphase neural network. That is, the embodiments described herein address the video frame synthesis task as a tensor recovery problem by reconstructing multiway tensors from a subset of slices (e.g., frontal slices). Due to the spatiotemporal correlation of video frames, the embodiments described herein can solve the tensor recovery problem by performing tensor rank minimization in the transform domain. The embodiments described herein can exploit the low-rank nature of video data in the transform domain with relatively high synthesis speeds.

[0023] The embodiments described herein can provide improvements over existing video synthesis techniques (e.g., video frame interpolation and prediction). For example, the embodiments described herein can improve the peak signal-to-noise ratio (PSNR) of interpolation and prediction by, for example, 4.13 dB and 4.16 dB. Additional benefits over existing video synthesis solutions include, for example, reduced motion blur and avoidance of artifacts.

[0024] Referring now to the drawings, in which like numerals represent the same or similar components, and initially to FIG. 1, a block diagram / flow chart showing a high level overview of a video frame composition system 100 is provided.

[0025] As shown, system 100 includes input video data 110 that includes multiple frames. For example, input video data 110 may include frame 112. Although input video data 110 is shown to include three frames, input video data 110 may include any suitable number of frames according to embodiments described herein. More specifically, input video data 110 includes one or more missing frames.

[0026] A primary goal of system 100 is to synthesize one or more missing video frames to generate restored video data (e.g., using video frame interpolation and / or prediction). To achieve this goal, system 100 further includes a transform-based tensor neural network (TTNet) 120.

[0027] To further explain the video frame synthesis task performed by system 100, the following definitions (1)-(5) are required. Given the reversible linear transform:

[0028]

number

[0029] is the tube multiplication of

[0030]

number

[0031]

number

[0032] can be defined as follows:

[0033]

number

[0034] Here, L -1 is the reciprocal of L,

[0035]

number

[0036] denotes the Hadamard (element-wise) product.

[0037]

number

[0038] and

[0039]

number

[0040] L product of

[0041]

number

[0042] is a tensor of size n1×n4×n3, so for i ∈ [n1] and j ∈ [n4],

[0043]

number

[0044] where each [n] corresponds to the index set {1,2,...,n}, and each of X(i,s,:) and Y(s,j,:) corresponds to a tube. For a tensor X, its transpose matrix

[0045]

number

[0046] is defined as follows:

[0047]

number

[0048] where k∈[n3] and (X (k) ) H is X (k) When X is L-diagonal, its (front) slices are the diagonal matrices and

[0049]

number

[0050] where

[0051]

number

[0052] is an L-diagonal with e on the diagonal, i.e., for i∈[n], the tube I(i,s,:)=L -1 (1") and all other tubes are 0", where

[0053]

number

[0054] indicates a tube with all entries equal to 1 or 0, respectively.

[0055]

number

[0056] The L-SVD of is defined as follows:

[0057]

number

[0058] where

[0059]

number

[0060] and

[0061]

number

[0062] is an L-orthogonal tensor,

[0063]

number

[0064] is an L-diagonal tensor. Then the rank of the L-tubes of T is defined to be the number of non-zero tubes in S. Finally, the tensor kernel norm of a tensor X is defined as

[0065]

number

[0066] where

[0067]

number

[0068] denotes the matrix kernel norm,

[0069]

number

[0070] is the frequency domain representation obtained by taking a Fourier transform along the third dimension of X.

[0071] Given the above definitions, the video frame synthesis task can be formally modeled as a tensor completion problem with random missing (frontal) slices. For example, consider the following video tensor:

[0072]

number

[0073] Let Ω⊆[n3] be the set of observed video frames. The observation tensor

[0074]

number

[0075] is the kth (frontal) slice

[0076]

number

[0077] can be defined as follows:

[0078]

number

[0079] where 1' denotes an n1 × n2 matrix of ones, and 0' denotes an n1 × n2 matrix of zeros. Then, the tensor

[0080]

number

[0081] The observed video frame of the video corresponding to

[0082]

number

[0083] can be expressed as

[0084]

number

[0085] The kth (frontal) slice of can be defined as follows:

[0086]

number

[0087] where

[0088]

number

[0089] denotes the Hadamard (element-wise) product again.

[0090] The tensor completion problem is generally solvable because compressive sensing (CS) theory shows that signals sampled at rates less than the Nyquist rate can be reconstructed, and the iterative shrinkage-threshold algorithm (ISTA) can be used to solve CS problems with dense matrix data.

[0091] With the foregoing in mind, as shown, TTNet 120 may include multiple phases 122-1 through 122-N, each having the same architecture but different parameter values.

[0092] The input video data 110 is the tensor X 0 and the phase 1 122-1 receives it. The tensor output X of the phase 1 122-1 1 is sent to the next phase of TTNet 120. The Nth phase of TTNet is sent to the previous phase X of TTNet 120. N-1 Receive the tensor output by N Outputs the tensor X N corresponds to the synthesis result, which is compared with the ground truth to calculate a training loss for convergence acceleration. A recovered result 130 having multiple frames, including frame 132, is obtained. The recovered result 130 corresponds to input video data that includes one or more missing frames.

[0093] Given the above definitions, the video frame synthesis task can be considered as completing a tensor T from the observed frontal slices indexed by Ω. The goal is then to find an image tensor X with L-tube rank ≦r such that

[0094]

number

[0095] More specifically, the tensor completion problem is presented as follows:

[0096]

number

[0097] Here, for some tensor T

[0098]

number

[0099] is defined as follows:

[0100]

number

[0101] and λ is a constant parameter.

[0102]

number

[0103] The term is used to provide sparsity. As will be explained in more detail below with reference to Figure 2, each of phases 122-1 through 122-N includes multiple components or modules designed to perform inference iterations of the Tensor ISTA algorithm used to solve the tensor completion problem. For t ∈ [1,N], it is proposed that the Tensor ISTA algorithm executed in phase t is as follows:

[0104]

number

[0105] where ρ is the step size and R t is the intermediate synthesis result in the original domain.

[0106] As will be described in more detail below with reference to Figure 2, a given one of the plurality of phases 122-1 to 122-N may include an update module, a transformation module, a soft threshold module, and an inverse transformation module. The update module may be calculated by using equation (9):

[0107]

number

[0108] Calculate the gradient of the intermediate synthesis result R in the original domain tThe transformation module transforms the intermediate synthesis result to obtain the transformed intermediate synthesis result in the transformation domain.

[0109]

number

[0110] and the soft threshold module is

[0111]

number

[0112] Applying a soft thresholding operation to each (frontal) slice of

[0113]

number

[0114] and an inverse transform module transforms the composite video data in the transform domain back to the original domain.

[0115] We can use a loss function that contains two loss terms: one that evaluates the accuracy of the synthesized video frame, and the other that accounts for the sparsity of the video frame tensor in the transform domain. For example, the accuracy loss term L accuracy and the sparsity loss term L sparsity The loss function L, which includes L=αL accuracy +βL sparsity (11) where α and β are parameters that balance the accuracy and sparsity loss terms. In an exemplary embodiment, α and β may be set to 1 and 0.1 by default, respectively. For example, L accuracy teeth

[0116]

number

[0117] and L sparsity teeth

[0118]

number

[0119] can be defined as follows:

[0120]

number

[0121] The tensor ISTA algorithm according to the embodiments described herein provides an improvement over conventional ISTA algorithms that convert video data into a matrix representation, which may lose spatiotemporal information.

[0122] Referring now to FIG. 2, a block diagram / flow chart is provided showing an overview of the phases of a transform-based tensor neural network (TTNet) 200.

[0123] As shown, the input tensor X generated in the t-1th phase t-1 210 is received by phase 220 of TTNet200, where

[0124]

number

[0125] Phase 220 corresponds to a given one of the N phases of TTNet described above with reference to FIG. 1. More specifically, phase 220 generates a tensor output X t is the t-th phase configured to generate 230, where

[0126]

number

[0127] Therefore, when t=N, X t 230 corresponds to the final tensor output of TTNet, X N is equal to.

[0128] Phase 220 returns the final output tensor X N , which corresponds to one inference iteration of the tensor iterative shrinkage threshold algorithm (ISTA) implemented by TTNet 200, used to generate . More specifically, phase 220 includes multiple modules configured to implement an inference iteration of the tensor ISTA algorithm, including an update module 222, a transform module 224, a soft-threshold module 226, and an inverse transform module 228.

[0129] As described in more detail above with reference to Figure 1, the update module 222 is configured to generate an updated intermediate synthesis result in the original domain. More specifically, the update module 222 generates an updated intermediate synthesis result in the original domain by using equation (9):

[0130]

number

[0131] We can calculate the gradient of

[0132]

number

[0133] , thereby generating an updated intermediate composite result in the original domain.

[0134] The goal of TTNet200 is to find a solution to the tensor completion problem in the trained transform domain and then transform the solution back to the original domain. To accomplish this, a transform module 224 is configured to convert updated intermediate composite results into transformed intermediate results in the transform domain. Further details regarding the transform module will now be described below with reference to FIG. 3.

[0135] 3, a block / flow diagram is provided illustrating an exemplary transform-based tensor neural network (TTNet) 300. As shown, TTNet 300 includes an update module 222 configured to receive tensor input 210, a transform module 224, and a soft-threshold module 226, as described above with reference to FIG.

[0136] As further shown, in this exemplary embodiment, the transform module 224 includes multiple components, including a first convolution (Conv) component 310-1, an activation function (AF) component 320, and a second Conv component 310-2. In one embodiment, the first and second Conv components 310-1 and 310-2 perform two-dimensional (2D) multi-channel convolution with different kernels, and the AF component 320 includes a rectified linear unit (ReLU) incorporating nonlinearity.

[0137] The first Conv component 310-1 receives the updated intermediate synthesis result from the update module 222 and performs Conv(R t ) is then generated by the AF component 320. t ) to apply an activation function. For example, when the AF component 320 includes ReLU, Conv(R t ) by applying an activation function to ReLU(Conv(R t Then, the second Conv component 310-2 generates a ReLU (Conv(Rt )) by applying a second convolution to

[0138]

number

[0139] which corresponds to the transformed intermediate composite result.

[0140] Referring again to Figure 2, the converted intermediate composite result

[0141]

number

[0142] After obtaining , the soft threshold module 226 is configured to apply a soft threshold based on the transformed intermediate synthesis result to generate synthesized video data in the transform domain. More specifically, the synthesized video data in the transform domain corresponds to a tensor. In one embodiment, the soft threshold module 226 uses

[0143]

number

[0144] 4, further details regarding the soft threshold module 226 will now be described below.

[0145] 4, a block / flow diagram is provided illustrating an exemplary transform-based tensor neural network (TTNet) 400. As shown, TTNet 400 includes the transform module 224, soft threshold module 226, and inverse transform module 228 described above with reference to FIG.

[0146] As further shown, in this exemplary embodiment, the soft threshold module 226 includes a plurality of soft threshold components 410-1 through 410-Q configured to perform parallel soft threshold operations, as described above with reference to FIG. 1, where Q is equal to n. More specifically, each of the plurality of soft threshold components 410-1 through 410-Q is configured to perform parallel soft threshold operations, as described above with reference to FIG. t ) (1) From F(R t ) (Q) as inputs and generate respective (matrix) outputs. After the parallel soft-threshold operations are performed, the outputs of the multiple soft-threshold components 410-1 to 410-Q are superimposed to generate the tensor

[0147]

number

[0148] which corresponds to the composite video data in the transform domain. For example, F(X t ) (k) =soft(F(R t ) (k) ,λ) (13) where soft(·) is the element-wise soft threshold operation defined as:

[0149]

number

[0150] and k∈[n3], where λ is a constant.

[0151] Referring again to FIG. 2, after the composite video data in the transform domain is generated, the inverse transform module 228 transforms the composite video data in the transform domain back to the original domain, X t230. Further details regarding the inverse transform module 228 will now be described below with reference to FIG.

[0152] 5, a block / flow diagram is provided illustrating an exemplary transform-based tensor neural network (TTNet) 500. As shown, TTNet 500 includes the soft-threshold module 226 described above with reference to FIG. 2 and an inverse transform module 228 configured to generate a tensor output 230.

[0153] As further shown, in this exemplary embodiment, the inverse transform module 228 includes multiple components, including a first convolution (Conv) component 510-1, an activation function (AF) component 520, and a second Conv component 510-2. In one embodiment, the first and second Conv components 510-1 and 510-2 perform two-dimensional (2D) multi-channel convolution with different kernels, and the AF component 520 includes a rectified linear unit (ReLU) incorporating nonlinearity. The network structure of the inverse transform module 228 is similar to that of the transform module 224, except that the Conv components have different kernel parameters.

[0154] The first Conv component 510-1 receives the composite video data in the transform domain from the soft threshold module 226 and:

[0155]

number

[0156] Then, the AF component 520 generates

[0157]

number

[0158] For example, when the AF component 520 includes a rectified linear unit (ReLU), the activation function is

[0159]

number

[0160] By applying

[0161]

number

[0162] Then, the second Conv component 510-2 performs the second convolution as

[0163]

number

[0164] Applying to

[0165]

number

[0166] Generates 230.

[0167] Referring again to Figure 2, the loss function may be used as described above with reference to Figure 1. More specifically, the loss function may be a linear combination of a first part that evaluates the accuracy of the synthetic video frame and a second part that provides sparsity. Further details regarding the loss function are described above with reference to Figure 1.

[0168] Referring to Figure 6, the final tensor output X N 6 shows exemplary pseudocode 600 describing the ISTA algorithm iterated from t=0 to t=N to generate

[0169] 7, a block diagram / flowchart illustrating a system / method 700 for implementing video frame synthesis using a transform-based tensor neural network (TTNet) is provided. More specifically, the system / method 700 may be used to interpolate and / or predict video frames. For example, the system / method 700 may be used to improve the peak signal-to-noise ratio (PSNR) of the interpolation and prediction (e.g., 4.13 dB and 4.16 dB), reduce motion blur, and avoid artifacts.

[0170] At block 710, input video data including one or more missing frames is received.

[0171] In block 720, the input video data is converted into an input tensor.

[0172] At block 730, output video data including one or more synthetic frames corresponding to the one or more missing frames is generated by using a transform-based tensor neural network (TTNet) including multiple phases that implement a tensor iterative shrinkage-threshold algorithm (ISTA) through tensor completion based on the input tensor. More specifically, each of the multiple phases may include multiple modules configured to perform one inference iteration of the tensor ISTA. Further details regarding the implementation of the inference iteration of the tensor ISTA will be described in more detail below with reference to FIG. 8.

[0173] At block 740, a loss function based on the output video data is obtained. In one embodiment, the loss function includes a first portion that evaluates the accuracy of one or more synthesized frames and a second portion that accounts for sparsity. More specifically, the loss function may be a linear combination of the first and second portions. For example, as described above with reference to FIG. 1, the loss function may be defined by equation (11).

[0174] Further details regarding blocks 710-740 of FIG. 7 are described above with reference to FIGS.

[0175] Referring now to FIG. 8, a block / flow diagram illustrating a system / method 800 for performing inference iterations of the Tensor Iterative Shrinkage Threshold Algorithm (ISTA) is provided.

[0176] In block 810, for a given one of multiple phases of a transform-based tensor neural network (TTNet), intermediate synthesis results in the original domain are updated. Further details regarding block 810 are described above with reference to FIG. 2.

[0177] In block 820, the intermediate synthesis result in the original domain is transformed into a transformed intermediate result in the transformed domain. More specifically, transforming the intermediate synthesis result in the original domain into a transformed intermediate result may include applying a first convolution to the intermediate synthesis result in the original domain, applying an activation function to the output of the first convolution, and applying a second convolution to the output of the activation function. In one embodiment, the first and second convolutions include two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU) incorporating nonlinearity. Further details regarding block 820 are described above with reference to FIGS. 2 and 3.

[0178] At block 830, a soft threshold based on the transformed intermediate compositing result is applied to generate composite image data in the transform domain. More specifically, the composite image data in the transform domain corresponds to a tensor. In one embodiment, applying a soft threshold based on the transformed intermediate compositing result to generate composite image data in the transform domain includes applying a plurality of soft threshold operations to each (frontal) slice of the transformed intermediate compositing result and overlapping the outputs of each of the plurality of soft threshold operations to form composite image data in the transform domain. Further details regarding block 830 are described above with reference to Figures 2 and 4.

[0179] In block 840, the synthetic video data in the transform domain is transformed back to the original domain using an inverse transform. More specifically, transforming the synthetic video data in the transform domain back to the original domain using an inverse transform may include applying a third convolution to the synthetic video data in the transform domain, applying a second activation function to the output of the third convolution, and applying a fourth convolution to the output of the second activation function. In one embodiment, the third and fourth convolutions include 2D multi-channel convolutions with different kernels, and the activation function is a ReLU that incorporates nonlinearity. The third and fourth convolutions may have different kernel parameters than the first and second convolutions of block 820. Further details regarding block 840 are described above with reference to Figures 2 and 5.

[0180] Further details regarding the ISTA algorithm are described above with reference to FIG.

[0181] Referring now to FIG. 9 , a block diagram illustrating an exemplary processing system 900 according to an embodiment of the present invention is provided. The processing system 900 includes a set of processing units (e.g., CPUs) 901, a set of GPUs 902, a set of memory devices 903, a set of communication devices 904, and a set of peripherals 905. The CPUs 901 may be single-core or multi-core CPUs. The GPUs 902 may be single-core or multi-core GPUs. The one or more memory devices 903 may include cache, RAM, ROM, and other memory (flash, optical, magnetic, etc.). The communication devices 904 may include wireless and / or wired communication devices (e.g., network (e.g., Wi-Fi, etc.) adapters, etc.). The peripherals 905 may include display devices, user input devices, printers, imaging devices, etc. The elements of the processing system 900 are connected by one or more buses or networks (collectively designated by reference numeral 910 in the drawings).

[0182] In some embodiments, memory device 903 may store specially programmed software modules configured to implement various aspects of the present invention. In some embodiments, special purpose hardware (e.g., application specific integrated circuits and field programmable gate arrays (FPGAs)) may be used to implement various aspects of the present invention.

[0183] In one embodiment, memory device 903 stores program code 906 for implementing a transform-based tensor neural network (TTNet). As described in more detail above, TTNet can be used to perform video frame synthesis (e.g., video frame interpolation and / or video frame prediction) in an improved manner.

[0184] Of course, as would be readily appreciated by one skilled in the art, processing system 900 may also include other elements (not shown) or omit certain elements. For example, as would be readily appreciated by one skilled in the art, various other input and / or output devices may be included in processing system 900, depending on the particular implementation of processing system 900. For example, various types of wireless and / or wired input and / or output devices may be included.

[0185] Although this disclosure includes a detailed description of cloud computing, it should be understood that practice of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0186] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0187] The characteristics are as follows:

[0188] On-demand self-service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, as needed automatically and without human interaction with the provider of the service.

[0189] Broad network access. Functionality is available across the network and accessed through standard mechanisms that facilitate use by a variety of thin and thick client platforms (e.g., cell phones, laptops, and PDAs).

[0190] Resource Pooling. To serve multiple consumers using a multi-tenant model, a provider's computing resources are pooled, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge over the exact location of the resources provided, although there is a sense of location independence in that they may be able to identify a location at a higher level of abstraction (e.g., country, state, or data center).

[0191] Rapid Elasticity. Capabilities can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited, and any amount can be purchased at any time.

[0192] Metered services. Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0193] The service model is as follows:

[0194] Software as a Service (SaaS). The functionality offered to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces, such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, except possibly for limited user-specific application configuration settings.

[0195] Platform as a Service (PaaS). The functionality offered to the consumer is the deployment of consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and possibly the application hosting environment configuration.

[0196] Infrastructure as a Service (IaaS). The functionality provided to the consumer is the provisioning of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating system, storage, deployed applications, and perhaps limited control over the selection of networking components (e.g., host firewalls).

[0197] The deployment model is as follows:

[0198] Private cloud: This cloud infrastructure is operated solely for one organization. It may be managed by that organization or a third party and may reside on-premises or off-premises.

[0199] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by those organizations or a third party and may reside on-premises or off-premises.

[0200] Public cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0201] Hybrid cloud: This cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain their own entities but are bound together by standard or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0202] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0203] Referring now to FIG. 10 , an exemplary cloud computing environment 1050 is shown. As shown, the cloud computing environment 1050 includes one or more cloud computing nodes 1010 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1054A, a desktop computer 1054B, a laptop computer 1054C, or an automotive computing system 1054N, or any combination thereof, may communicate. The nodes 1010 may also communicate with each other. These nodes may be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or any combination thereof. This enables the cloud computing environment 1050 to provide infrastructure, platform, or software, or any combination thereof, as a service for which cloud consumers are not required to maintain resources on their local computing devices. It is understood that the types of computing devices 1054A-N shown in FIG. 10 are intended to be exemplary only, and that the computing node 1010 and cloud computing environment 1050 can communicate with any type of computing device through any type of network or network-addressable connection (e.g., using a web browser) or both.

[0204] Referring now to Figure 11, a set of functional abstraction layers provided by cloud computing environment 1050 (Figure 10) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 11 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0205] Hardware and software layer 1160 includes hardware and software components. Examples of hardware components include mainframe 1161, RISC (Reduced Instruction Set Computer) architecture-based server 1162, server 1163, blade server 1164, storage device 1165, and network and network forming components 1166. In some embodiments, software components include network application server software 1167 and database software 1168.

[0206] Virtualization layer 1170 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 1171, virtual storage 1172, virtual networks including virtual private networks 1173, virtual applications and operating systems 1174, and virtual clients 1175.

[0207] In one example, management layer 1180 may provide the following functions: Resource provisioning 1181 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1182 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 1183 provides access to the cloud computing environment for consumers and system administrators. Service level management 1184 provides allocation and management of cloud computing resources to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 1185 provides advance arrangements for and procurement of cloud computing resources where future demand is predicted by SLAs.

[0208] The workload layer 1190 provides examples of functions for which a cloud computing environment may be used. Examples of workloads and functions that may be provided from this layer include mapping and navigation 1191, software development and lifecycle management 1192, virtual classroom instruction delivery 1193, data analytics processing 1194, transaction processing 1195, and video compositing 1196.

[0209] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0210] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: Portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves with recorded instructions, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed to refer to transitory signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0211] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0212] Computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by using state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0213] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0214] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, cause the computer to implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored includes instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0215] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0216] References herein to "one embodiment" or "an embodiment" of the invention, and other variations thereof, mean that the particular features, structures, characteristics, etc. described in connection with that embodiment are included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," and any other variations thereof, in various places throughout the specification are not necessarily all referring to the same embodiment.

[0217] It will be understood that the use of any of the following " / ," "and / or," and "at least one," such as in cases such as "A / B," "A and / or B," and "at least one of A and B," is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in cases of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This may be expanded or enumerated in many ways, as will be readily apparent to those skilled in this and related arts.

[0218] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, may be executed concurrently, may be executed substantially concurrently in a partially or fully overlapping manner, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. In addition, it will be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0219] As used herein, the terms "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or a combination thereof, working together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, or a separate processor- or computing element-based controller (e.g., logic gates, etc.), or a combination thereof. A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.), which may be on or off-board or dedicated for use by the hardware processor subsystem.

[0220] In some embodiments, a hardware processor subsystem may include and execute one or more software elements, which may include an operating system, one or more applications, or specific code, or any combination thereof, to achieve a specified result.

[0221] In other embodiments, a hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result, which may include one or more application-specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0222] These and other variations of the hardware processor subsystem are also contemplated by embodiments of the present invention.

[0223] Having described a preferred embodiment of video compositing (intended to be illustrative and not limiting), it is noted that modifications and variations will occur to those skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments disclosed that are within the scope of the invention as outlined by the appended claims. With the aspects of the invention thus described, what is claimed and desired to be protected by Letters Patent is set forth in the appended claims, having the particularity particularly required by the Patent Laws.

Claims

1. 1. A method for implementing video frame synthesis using a tensor neural network, comprising: receiving input video data including one or more missing frames; converting the input video data into an input tensor; generating output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including multiple phases that implements an iterative tensor shrinkage and threshold algorithm (ISTA) through tensor completion based on the input tensor; obtaining a loss function based on the output video data; A method comprising:

2. A computer program product for causing a computer to carry out the method of claim 1.

3. A computer-readable storage medium having the computer program according to claim 2 recorded thereon.

4. A system for realizing video frame synthesis using a tensor neural network, comprising: a memory configured to store program code; and at least one processor device operatively coupled with the memory, the at least one processor device comprising: receiving input video data including one or more missing frames; converting the input video data into an input tensor; generating output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including multiple phases that implements an iterative tensor shrinkage and threshold algorithm (ISTA) through tensor completion based on the input tensor; obtaining a loss function based on the output video data; a system configured to execute program code stored in the memory to perform the steps of:

Images