Method, system, and program product for video frame synthesis using tensor neural networks

Abstract

A method for implementing video frame synthesis using a tensor neural network includes receiving input video data including one or more missing frames; converting the input video data into an input tensor; generating output video data through a transform-based tensor neural network (TTNet) through tensor completion based on the input tensor, the output video data including one or more synthesized frames corresponding to the one or more missing frames, the TTNet including multiple stages of implementing a tensor iterative shrinkage thresholding algorithm (ISTA); and obtaining a loss function based on the output video data.

Description

Background Art

[0001] The present invention relates generally to artificial intelligence and, more particularly, to video frame synthesis using tensor neural networks.

[0002] Video frame synthesis is a task in computer vision that has attracted great interest in a wide variety of applications. During video frame synthesis, new video frames are constructed from existing videos. Video frame synthesis can be divided into two categories, video frame interpolation and video frame prediction. Video frame interpolation constructs new video frames by restoring missing frames between existing video frames, and video frame prediction constructs new video frames by generating future frames from previous frames. Video frame synthesis techniques have a variety of uses. For example, video frame synthesis techniques can transform standard videos into high-quality slow-motion videos with higher frame rates and smooth view transitions, compensate for distortion from camera shake in video recording using video stabilization, and alleviate the missing video frame problem in wireless video transmission using motion smoothing, for example, in unmanned aerial vehicles (UAVs) and virtual reality (VR) applications.

[0003] Existing video frame synthesis solutions have one or more deficiencies. For example, existing neural network-based methods do not explicitly impose low tensor levels to capture the spatiotemporal correlations of video frames in high-dimensional space, and there is no underlying mathematical interpretation. In addition, iterative algorithms based on compressed sensing typically require manual parameters and relatively long runtimes for high-quality video frame synthesis. For example, some existing optimization algorithms may take more than an hour to process a one-minute video (30 frames per second) with 100 missing frames. Therefore, it would be advantageous to provide a video synthesis solution that at least addresses the above-mentioned issues. Summary of the Invention

[0004] A method for implementing video frame synthesis using a tensor neural network, comprising: receiving input video data including one or more missing frames; converting the input video data into an input tensor; generating output video data by using a transform-based tensor neural network (TTNet) based on a tensor completion of the input tensor, the output video data including one or more synthesized frames corresponding to the one or more missing frames, the TTNet including multiple stages of implementing a tensor iterative shrinkage thresholding algorithm (ISTA); and obtaining a loss function based on the output video data.

[0005] In an embodiment of the method, generating an output tensor includes: for a given stage among a plurality of stages of the TTNet, updating an intermediate synthesis result in the original domain, transforming the intermediate synthesis result in the original domain into a transformed intermediate synthesis result in a transform domain, applying soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transform domain, and transforming the synthesized video data in the transform domain back to the original domain using an inverse transform.

[0006] In an embodiment of the method, an intermediate synthesis result in the original domain is defined based in part on a video tensor and an observation tensor received by a given stage of the plurality of stages.

[0007] In an embodiment of the method, transforming the intermediate synthesis result in the original domain into the transformed intermediate synthesis result includes: applying a first convolution to the intermediate synthesis result, applying an activation function to the output of the first convolution, and applying a second convolution to the output of the activation function to generate a transformation. In one embodiment, the first and second convolutions include two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU).

[0008] In an embodiment of the method, applying soft thresholding based on the transformed intermediate synthesis result to generate synthetic video data in the transformed domain includes applying multiple soft thresholding operations in parallel to each front slice of the transformed intermediate synthesis result, and stacking the outputs of the multiple soft thresholding operations to generate synthetic video data in the transformed domain.

[0009] In an embodiment of the method, transforming the synthesized video data in the transformed domain back to the original domain using an inverse transform includes: applying a third convolution to the synthesized video data in the transformed domain, applying a second activation function to the output of the third convolution, and applying a fourth convolution to the output of the second activation function. In one embodiment, the third and fourth convolutions are 2D multi-channel convolutions with different kernels, and the activation function is ReLU.

[0010] In an embodiment of the method, transforming the synthesized video data in the transformed domain back to the original domain using an inverse transform comprises: applying a first convolution to the synthesized video data in the transformed domain, applying an activation function to the output of a third convolution, and applying a second convolution to the output of the activation function. In one embodiment, the first and second convolutions comprise 2D multi-channel convolutions with different kernels, and the activation function is a ReLU.

[0011] In an embodiment of the method, the loss function comprises a first part that evaluates the accuracy of the one or more synthesized frames, and a second part that imposes sparsity. More specifically, the loss function may be a linear combination of the first and second parts.

[0012] A system for implementing video synthesis using a tensor neural network includes a memory configured to store program code and at least one processor device operably coupled to the memory. The at least one processor device is configured to execute the program code stored on the memory device to receive input video data including one or more missing frames, convert the input video data into an input tensor, perform tensor completion based on the input tensor, generate output video data using a transform-based tensor neural network (TTNet), the output video data including one or more synthesized frames corresponding to the one or more missing frames, the TTNet including multiple stages implementing a tensor iterative shrinkage thresholding algorithm (ISTA), and obtain a loss function based on the output video data.

[0013] In one embodiment of the system, at least one processor device is configured to generate an output tensor by, for a given one of a plurality of stages of the TTNet, updating an intermediate synthesis result in the original domain, transforming the intermediate synthesis result in the original domain into a transformed intermediate synthesis result in a transformed domain, applying soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transformed domain, and transforming the synthesized video data in the transformed domain back to the original domain using an inverse transform.

[0014] In an embodiment of the system, an intermediate synthesis result in the original domain is defined based in part on a video tensor and an observation tensor received by a given one of the plurality of stages.

[0015] In an embodiment of the system, at least one processor device is configured to transform the intermediate synthesis result in the original domain into a transformed intermediate synthesis result by applying a first convolution to the intermediate synthesis result, applying an activation function to an output of the first convolution, and applying a second convolution to an output of the activation function to produce a transformation. In one embodiment, the first and second convolutions comprise two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU).

[0016] In an embodiment of the system, at least one processor device is configured to apply soft thresholding based on the transformed intermediate synthesis result to generate synthetic video data in the transform domain by applying multiple soft thresholding operations in parallel to each previous slice of the transformed intermediate synthesis result, and stacking the outputs of the multiple soft thresholding operations to produce synthetic video data in the transform domain.

[0017] In an embodiment of the system, at least one processor device is configured to transform the synthesized video data in the transformed domain back to the original domain using an inverse transform, comprising applying a first convolution to the synthesized video data in the transformed domain, applying an activation function to an output of a third convolution, and applying a second convolution to an output of the activation function. In one embodiment, the first convolution and the second convolution comprise 2D multi-channel convolutions with different kernels, and the activation function is a ReLU.

[0018] In an embodiment of the system, the loss function comprises a first part that evaluates the accuracy of the one or more synthesized frames, and a second part that imposes sparsity. More specifically, the loss function may be a linear combination of the first and second parts.

[0019] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following description will provide details of the preferred embodiments with reference to the following drawings, in which:

[0021] Figure 1 is a block diagram / flowchart of a high-level overview of a video frame synthesis system using a transform-based tensor neural network (TTNet) according to an embodiment of the present invention;

[0022] Figure 2 According to an embodiment of the present invention Figure 1 The block diagram / flowchart of the modules of TTNet stages;

[0023] Figure 3 According to an embodiment of the present invention Figure 2 A block diagram / flow chart of an exemplary transformation module of the stage;

[0024] Figure 4 According to an embodiment of the present invention Figure 2 A block diagram / flow chart of an exemplary soft thresholding module for the stage of FIG.

[0025] Figure 5 According to an embodiment of the present invention Figure 2 A block diagram / flow chart of an example inverse transform module for the stage of FIG.

[0026] Figure 6 is an exemplary pseudo code describing an iterative tensor shrinkage thresholding algorithm (ISTA) according to an embodiment of the present invention;

[0027] Figure 7 is a block diagram / flow chart of a system / method for implementing video frame synthesis using a transform-based tensor neural network (TTNet) implementing an iterative tensor shrinkage thresholding algorithm (ISTA) according to an embodiment of the present invention;

[0028] Figure 8 According to an embodiment of the present invention, Figure 7 A block diagram / flow chart of a system / method for inference iteration of Tensor ISTA;

[0029] Figure 9 is a block diagram of a processing system according to an embodiment of the present invention;

[0030] Figure 10 is a block diagram showing an illustrative cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to one embodiment; and

[0031] Figure 11 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to one embodiment. DETAILED DESCRIPTION

[0032] Embodiments of the present invention use tensor neural networks to perform video synthesis. More specifically, the embodiments described herein may represent video data as tensors and construct a transform-based tensor neural network (TTNet) by expanding the tensor iterative shrinkage thresholding algorithm (ISTA) into a multi-stage neural network. That is, the embodiments described herein regard the video frame synthesis task as a tensor recovery problem by reconstructing a multi-way tensor from a subset of slices (e.g., the front slice). Due to the spatiotemporal correlation of video frames, the embodiments described herein may solve the tensor recovery problem by performing tensor rank minimization in the transform domain. The embodiments described herein may exploit the low dimensionality of video data with a relatively high synthesis speed in the transform domain.

[0033] The embodiments described herein can provide improvements to existing video synthesis techniques (e.g., interpolation and prediction of video frames). For example, the embodiments described herein can improve the peak signal-to-noise ratio (PSNR) of interpolation and prediction by, for example, 4.13 dB and 4.16 dB. Other advantages over existing video synthesis solutions include, for example, motion blur reduction and artifact avoidance.

[0034] Referring now to the drawings, in which like numerals represent the same or similar elements, and first to Figure 1 , provides a block diagram / flowchart showing a high-level overview of a video frame synthesis system 100.

[0035] As shown, system 100 includes input video data 110 having a plurality of frames. For example, input video data 110 may include frame 112. Although input video data 110 is shown as including three frames, input video data 110 may include any suitable number of frames according to the embodiments described herein. More specifically, input video data 110 includes one or more missing frames.

[0036] The primary goal of the system 100 is to synthesize one or more missing video frames to generate recovered video data (eg, using video frame interpolation and / or prediction). To achieve this goal, the system 100 also includes a transform-based tensor neural network (TTNet) 120 .

[0037] The following definitions (1) to (5) will be needed to further describe the video frame synthesis task performed by the system 100. Given a reversible linear transformation Pipe multiplication It can be defined as:

[0038]

[0039] Among them L -1 Is the reciprocal of L, ☉ represents the Hadamard (element-wise) product. Tensor and L product is a tensor of size n1×n4×n3 such that:

[0040]

[0041] For i∈[n1] and j∈[n4], where each [n] corresponds to the index set {1, 2, ..., n}, and X(i, s, :) and Y(s, j, :) each correspond to a tube. For a tensor X, its transpose is defined as:

[0042]

[0043] where k∈[n3] and (X (k) ) H Represents X (k) The Hermitian (conjugate) transpose of . If X is L-diagonal, then its (positive) slices correspond to diagonal matrices, and in is L diagonal, e is on the diagonal. That is, for i∈[n], I(i, s, :) = L -1 (1″), and all other tubes are 0″, where Represents a tube where all entries are equal to 1 or 0 respectively. The L-SVD is defined as:

[0044]

[0045] in and is an L-orthogonal tensor and is an L-diagonal tensor. Then, the L-tube rank of T is defined as the number of non-zero tubes of S. Finally, the tensor nuclear norm of the tensor X is defined as

[0046]

[0047] where ||·|| * represents the matrix nuclear norm, is the frequency domain representation obtained by Fourier transforming along the third dimension of X.

[0048] Given the above definition, the video frame synthesis task can be formally modeled as a tensor completion problem with randomly missing (front) slices. For example, consider the video tensor make is the set of observed video frames. can be defined as having the kth (front) slice Φ(:,:,k) as follows:

[0049]

[0050] Where 1' represents an n1×n2 matrix of 1s, and 0' represents an n1×n2 matrix of 0s. Then, corresponding to the tensor The observed video frame of a video can be represented as Y = Φ⊙T and the kth (front) slice of Y(:,:,k) can be defined as:

[0051]

[0052] where ☉ again denotes the Hadamard (element-wise) product.

[0053] Since compressed sensing (CS) theory shows that signals sampled at rates lower than the Nyquist rate can be reconstructed, and the iterative shrinkage thresholding algorithm (ISTA) can be used to solve CS problems with dense matrix data, tensor completion problems can generally be solved.

[0054] Considering the above, as shown in the figure, TTNet 120 may include a plurality of stages 122 - 1 to 122 -N. Each of the stages 122 - 1 to 122 -N has the same architecture but different parameter values.

[0055] The input video data 110 is converted into a tensor X 0 , which is received by stage 1 122-1. In stage 1 122-1, X 1 The tensor output at is sent to the next stage of TTNet 120. The Nth stage of TTNet receives the tensor output by the previous stage X of TTNet120. N－1 Output tensor, and output tensor X N Tensor X NA synthesized result corresponding to the training loss to be compared with the true value to calculate convergence acceleration is obtained. A restored result 130 having a plurality of frames including frame 132 is obtained. The restored result 130 corresponds to input video data including one or more missing frames.

[0056] Considering the above definition, the video frame synthesis task can be cast as completing the tensor T from the observed previous slice indexed by Ω. Therefore, the goal becomes to find a video tensor X with L tube rank ≤ r such that Φ⊙X=Y. More specifically, the tensor completion problem is given as follows:

[0057]

[0058] For some tensors is defined as and λ is a constant parameter. The term λ||X|| TNN Used to enforce sparsity. Figure 2 Described in more detail, each of stages 122-1 to 122-N includes multiple components or modules designed to implement an inference iteration of the tensor ISTA algorithm for solving the tensor completion problem. For t∈[1,N], the tensor ISTA algorithm executed at stage t is proposed as follows:

[0059]

[0060] R t =X t-1 -ρΦ☉(Φ☉X t-1 -Y) (10)

[0061] where ρ is the step size, R t is the intermediate synthesis result in the original domain.

[0062] As will be referred to below Figure 2 As further described in detail, a given stage in the plurality of stages 122-1 to 122-N may include an update module, a transform module, a soft thresholding module, and an inverse transform module. The update module calculates The gradient of , to update the original domain R t The intermediate synthesis result in the transform module transforms the intermediate synthesis result into the transform domain The transformed intermediate synthesis result in . The soft threshold processing module Apply a soft thresholding operation to each (front) slice of to generate the transform domain The inverse transform module transforms the synthesized video data in the transform domain back to the original domain.

[0063] A loss function including two loss terms can be used. One loss term evaluates the accuracy of the synthesized video frame, while the other loss term enforces the sparsity of the video frame tensor in the transform domain. For example, including the accuracy loss term L accuracy and the sparsity loss term L sparsity The loss function L can be defined as:

[0064] L=αL accuracy +βL sparsity (11)

[0065] Where α and β are parameters that balance the accuracy and sparsity loss terms. In the illustrative embodiment, α and β can be set to 1 and 0.1 respectively by default. For example, L accuracy Can be defined as And L sparsity Can be defined as Make

[0066]

[0067] The tensor ISTA algorithm according to the embodiments described herein provides an improvement over traditional ISTA algorithms, which transform video data into a matrix representation that may lack spatiotemporal information.

[0068] Now refer to Figure 2 , provides a block diagram / flowchart illustrating an overview of the stages of a transform-based tensor neural network (TTNet) 200.

[0069] As shown in the figure, in the t-1 stage X t－1 At 210, the input tensor is generated, where Received by stage 220 of TTNet 200. Stage 220 corresponds to a given stage in the plurality of N stages of TTNet, as described above with reference to Figure 1 More specifically, stage 220 is configured to generate a tensor output X t 230th stage, where Therefore, if t = N, then X t 230 equals X N , corresponding to the final tensor output of TTNet.

[0070] Stage 220 corresponds to one inference iteration of the Iterative Shrinkage Thresholding Algorithm (ISTA) implemented by TTNet 200, which is used to generate the final output tensor X N More specifically, stage 220 includes a plurality of modules configured to implement an inference iteration of the tensor ISTA algorithm, including an update module 222 , a transform module 224 , a soft thresholding module 226 , and an inverse transform module 228 .

[0071] The update module 222 is configured to generate an updated intermediate synthesis result in the original domain, as described above with reference to Figure 1 More specifically, the update module 222 can calculate the value of The gradient of R t =X t-1 -ρΦ⊙(Φ⊙X t-1 -Y), to generate updated intermediate synthetic results in the original domain.

[0072] The purpose of TTNet 200 is to find a solution to the tensor completion problem in the learned transformed domain and then transform the solution back to the original domain. To achieve this, the transformation module 224 is configured to transform the updated intermediate synthesis result into the transformed intermediate result in the transformed domain. Figure 3 Further details about the transformation module are described.

[0073] refer to Figure 3 , provides a block diagram / flowchart illustrating an exemplary transformation-based tensor neural network (TTNet) 300. As shown, TTNet 300 includes an update module 222 configured to receive a tensor input 210, a transformation module 224, and the above reference Figure 2 The soft thresholding processing module 226 is described.

[0074] As further shown, in this illustrative example, the transformation module 224 includes multiple components, including a first convolution (Conv) component 310-1, an activation function (AF) component 320, and a second Conv component 310-2. In one embodiment, the first Conv component 310-1 and the second Conv component 310-2 implement two-dimensional (2D) multi-channel convolutions with different kernels, and the AF component 320 includes a rectified linear unit (ReLU) incorporating nonlinearity.

[0075] The first Conv component 310-1 receives the updated intermediate synthesis result from the update module 222 to generate Conv(R t ). Then, the AF component 320 applies the activation function to Conv(R t ). For example, if the AF component 320 includes ReLU, the activation function is applied to Conv(R t )Generate ReLU(Conv(R t )). Then, the second Conv component 310-2 pairs ReLU (Conv (R t ))Apply the second convolution to generate This corresponds to transforming the intermediate synthesis result.

[0076] Return Reference Figure 2, synthesize the results in the middle of the transformation Afterwards, a soft thresholding module 226 is obtained, which is configured to apply soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transformed domain. More specifically, the synthesized video data in the transformed domain corresponds to a tensor. In one embodiment, the soft thresholding module 226 is configured to apply soft thresholding to the transformed intermediate synthesis result. Each (front) slice of is subjected to a soft thresholding operation to generate synthetic video data in the transform domain. Figure 4 Further details regarding the soft thresholding module 226 are described.

[0077] refer to Figure 4 , provides a block diagram / flowchart showing an exemplary transformation-based tensor neural network (TTNet) 400. As shown in the figure, TTNet 400 includes the above reference Figure 2 The transform module 224, the soft thresholding module 226 and the inverse transform module 228 are depicted.

[0078] As further shown, in this illustrative example, the soft thresholding module 226 includes a plurality of soft thresholding components 410-1 through 410-Q, which are configured to perform parallel soft thresholding operations, where Q is equal to n3, as described above with reference to Figure 1 More specifically, each of the plurality of soft thresholding components 410-1 to 410-Q is configured to receive F(R t ) (1) to F(R t ) (Q) After performing the parallel soft thresholding operation, the outputs of the plurality of soft thresholding components 410-1 to 410-Q are stacked together to form a tensor Corresponds to the synthesized video data in the transform domain. For example:

[0079] F(X t ) (k) =soft(F(R t ) (k) ,λ) (13)

[0080] where soft(·) is an element-wise soft thresholding operation defined as follows:

[0081]

[0082] and where k∈[n3] and λ is a constant.

[0083] Return Reference Figure 2After the synthesized video data in the transform domain has been generated, the inverse transform module 228 is configured to transform the synthesized video data in the transform domain back to the original domain to generate a video image corresponding to X t 230 in the original domain of the synthesized video data. Figure 5 Further details regarding the inverse transform module 228 are described.

[0084] refer to Figure 5 , provides a block diagram / flow chart showing an exemplary transform-based tensor neural network (TTNet) 500. As shown in the figure, TTNet 500 includes a soft thresholding processing module 226 and an inverse transform module 228, which are configured to generate the above reference Figure 2 The tensor output 230 is described.

[0085] As further shown, in this illustrative example, the inverse transform module 228 includes multiple components, including a first convolution (Conv) component 510-1, an activation function (AF) component 520, and a second Conv component 510-2. In one embodiment, the first Conv component 510-1 and the second Conv component 510-2 implement two-dimensional (2D) multi-channel convolutions with different kernels, and the AF component 520 includes a rectified linear unit (ReLU) incorporated with nonlinearity. The network structure of the inverse transform module 228 is similar to that of the transform module 224, except that their Conv components have different kernel parameters.

[0086] The first Conv component 510-1 receives the synthesized video data in the transform domain from the soft thresholding module 222 to generate The AF component 520 then applies the activation function to For example, if the AF component 520 includes a rectified linear unit (RELU), the activation function is applied to generate Then, the second convolution component 510-2 applies the second convolution to To generate

[0087]

[0088] Return Reference Figure 2 , you can use the reference above Figure 1 More specifically, the loss function can be a linear combination of the first part that evaluates the accuracy of the synthesized video frame and the second part that imposes sparsity. Figure 1 Further details about the loss function are described.

[0089] refer to Figure 6An exemplary pseudocode 600 is shown describing the ISTA algorithm that iterates from t=0 to t=N to generate the final tensor output X N .

[0090] Now refer to Figure 7 , a block diagram / flowchart illustrating a system / method 700 for implementing video frame synthesis using a transform-based tensor neural network (TTNet) is provided. More specifically, the system / method 700 can be used to perform interpolation and / or prediction of video frames. For example, the system / method 700 can be used to improve the peak signal-to-noise ratio (PSNR) of interpolation and prediction (e.g., 4.13 dB and 4.16 dB), reduce motion blur, and avoid artifacts.

[0091] At block 710 , input video data including one or more missing frames is received.

[0092] At block 720 , input video data is converted into an input tensor.

[0093] At block 730, a transform-based tensor neural network (TTNet) is used to generate output video data by performing tensor completion based on the input tensor, the output video data including one or more synthesized frames corresponding to the one or more missing frames, the TTNet including a plurality of stages that implement a tensor iterative shrinkage thresholding algorithm (ISTA). More specifically, each of the plurality of stages may include a plurality of modules that are configured to implement an inference iteration of the tensor ISTA. More details on implementing the inference iteration of the tensor ISTA will be provided below with reference to Figure 8 Describe in more detail.

[0094] At block 740, a loss function is obtained based on the output video data. In one embodiment, the loss function includes a first part that evaluates the accuracy of one or more synthesized frames, and a second part that imposes sparsity. More specifically, the loss function can be a linear combination of the first and second parts. For example, the loss function can be defined by Equation (11), as described above with reference to Figure 1 As stated.

[0095] Reference above Figures 1 to 6 Describes about Figure 7 Further details of blocks 710 to 750 are provided.

[0096] Now refer to Figure 8 , a block diagram / flow diagram illustrating a system / method 800 for implementing an inference iteration of a tensor iterative shrinkage thresholding algorithm (ISTA) is provided.

[0097] At block 810, for a given one of a plurality of stages of a transform-based tensor neural network (TTNet), an intermediate synthesis result in the original domain is updated. Figure 2 Further details regarding block 810 are described.

[0098] At block 820, the intermediate synthesis result in the original domain is transformed into a transformed intermediate result in the transformed domain. More specifically, transforming the intermediate synthesis result in the original domain into the transformed intermediate result may include applying a first convolution to the intermediate synthesis result in the original domain, applying an activation function to the output of the first convolution, and applying a second convolution to the output of the activation function. In one embodiment, the first convolution and the second convolution include two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (RELU) combined with a nonlinearity. Figure 2 and 3 Further details regarding block 820 are described.

[0099] At block 830, soft thresholding is applied based on the transformed intermediate composite result to generate composite video data in the transform domain. More specifically, the composite video data in the transform domain corresponds to a tensor. In one embodiment, applying soft thresholding based on the transformed intermediate composite result to generate composite video data in the transform domain includes applying a plurality of soft thresholding operations to each (front) slice of the transformed intermediate composite result, and superimposing the output of each of the plurality of soft thresholding operations to form composite video data in the transform domain. Figure 2 and 4 Further details regarding block 830 are described.

[0100] At block 840, the composite video data in the transform domain is transformed back to the original domain using an inverse transform. More specifically, transforming the composite video data in the transform domain back to the original domain using an inverse transform may include applying a third convolution to the composite video data in the transform domain, applying a second activation function to the output of the third convolution, and applying a fourth convolution to the output of the third activation function. In one embodiment, the third and fourth convolutions include 2D multi-channel convolutions with different kernels, and the activation function is a recursive convolution combined with a nonlinearity. The third and fourth convolutions may have different kernel parameters than the first and second convolutions of block 820. Figure 2 and 5 Further details regarding block 840 are described.

[0101] Reference above Figure 6 Further details about the ISTA algorithm are described.

[0102] Now refer to Figure 9, a block diagram illustrating an exemplary processing system 900 according to an embodiment of the present invention is provided. The processing system 900 includes a processing unit set (e.g., CPU) 901, a GPU set 902, a memory device set 903, a communication device set 904, and a peripheral device set 905. CPU 901 can be a single-core or multi-core CPU. GPU 902 can be a single-core or multi-core GP. One or more memory devices 903 can include cache, RAM, ROM, and other memories (flash memory, optical, magnetic, etc.). Communication device 904 can include wireless and / or wired communication devices (e.g., network (e.g., WIFI, etc.) adapters, etc.). Peripheral devices 905 can include display devices, user input devices, printers, imaging devices, etc. The elements of the processing system 900 are connected via one or more buses or networks (collectively represented by reference numeral 910).

[0103] In an embodiment, the memory device 903 may store a specially programmed software module configured to implement various aspects of the present invention. In an embodiment, dedicated hardware (e.g., an application specific integrated circuit, a field programmable gate array (FPGA), etc.) may be used to implement various aspects of the present invention.

[0104] In an embodiment, storage device 903 stores program code for implementing a transform-based tensor neural network (TTNet) 906. As described in further detail above, TTNet can be used to perform video frame synthesis (e.g., video frame interpolation and / or video frame prediction) in an improved manner.

[0105] Of course, the processing system 900 may also include other elements (not shown) that are readily apparent to those skilled in the art, and may omit certain elements. For example, as will be readily understood by those skilled in the art, various other input devices and / or output devices may be included in the processing system 900, including various types of wireless and / or wired input and / or output.

[0106] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0107] Cloud computing is a service delivery model that provides convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0108] Features are as follows:

[0109] On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capabilities, such as server time and network storage, as needed without manual interaction with the service provider.

[0110] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0111] Resource pooling: Pooling a provider's computing resources to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence is present in the sense that consumers typically do not have control or knowledge of the exact location of the provisioned resources, but are instead able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0112] Rapid elasticity: In some cases, capacity can be automatically and rapidly and elastically provisioned to scale up and down quickly. To the consumer, the capacity available for provisioning typically appears unlimited and can be purchased at any time in any quantity.

[0113] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0114] The service model is as follows:

[0115] Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0116] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications they create or acquire onto cloud infrastructure. Applications are built using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and possibly the configuration of the application hosting environment.

[0117] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0118] The deployment model is as follows:

[0119] Private cloud: Cloud infrastructure is operated only for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0120] Community cloud: Cloud infrastructure is shared by multiple organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0121] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0122] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that are maintained as unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0123] The cloud computing environment is service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0124] Now refer to Figure 10 , depicts an illustrative cloud computing environment 1050. As shown, the cloud computing environment 1050 includes one or more cloud computing nodes 1010, and local computing devices used by cloud consumers such as cloud consumers (PDAs) or cell phones 1054A, desktop computers 1054B, laptop computers 1054C and / or automobile computer systems 1054N can communicate with these cloud computing nodes 1010. The nodes 1010 can communicate with each other. They can be physically or virtually grouped in one or more networks (not shown), one or more networks such as the private clouds, community clouds, public clouds or hybrid clouds described above, or a combination thereof. This allows the cloud computing environment 1050 to provide infrastructure, platforms and / or software as services without the cloud consumer needing to maintain resources on local computing devices. It should be understood that Figure 2The types of computing devices 1054A- 1054N shown are intended to be illustrative only, and computing node 1010 and cloud computing environment 1050 may communicate with any type of computerized device over any type of network and / or network-addressable connection (eg, using a web browser).

[0125] Now refer to Figure 11 , showing the cloud computing environment 1050 ( Figure 10 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 11 The components, layers, and functions shown in the figure are for illustration only and the embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0126] Hardware and software layer 1160 includes hardware and software components. Examples of hardware components include: mainframe 1161; servers based on RISC (Reduced Instruction Set Computer) architecture 1162; servers 1163; blade servers 1164; storage devices 1165; and network and networking components 1166. In some embodiments, software components include network application server software 1167 and database software 1168.

[0127] The virtualization layer 1170 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 1171 ; virtual storage 1172 ; virtual networks 1173 , including virtual private networks; virtual applications and operating systems 1174 ; and virtual clients 1175 .

[0128] In one example, the management layer 1180 may provide the functionality described below. Resource provisioning 1181 provides dynamic acquisition of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 1182 provides cost tracking when utilizing resources within a cloud computing environment, as well as accounting or invoicing for those resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 1183 provides access to the cloud computing environment for consumers and system administrators. Service level management 1184 provides allocation and management of cloud computing resources so that the required service levels are met. Service level agreement (SLA) planning and implementation 1185 provides pre-arrangement and procurement of cloud computing resources in anticipation of future demand for cloud computing resources based on SLAs.

[0129] The workload layer 1190 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 1191; software development and lifecycle management 1192; virtual classroom education delivery 1193; data analytics processing 1194; transaction processing 1195; and video synthesis 1196.

[0130] The present invention may be a system, method and / or computer program product at any possible level of integrated technical detail. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention.

[0131] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device, a punched card or raised structure such as a slot having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0132] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0133] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit comprising, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be configured to execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform various aspects of the present invention.

[0134] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing device to produce a machine, such that the instructions, executed via the processor of the computer or other programmable data processing device, create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct the computer, programmable data processing device, and / or other device to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture including instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device, thereby producing a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0137] Reference throughout the specification to "one embodiment" or "an embodiment" of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" and any other variations thereof in various places throughout the specification are not necessarily all referring to the same embodiment.

[0138] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of " / ," "and / or," and "at least one of" is intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the case of "A, B and / or C" and "at least one of A, B, and C," such phrases are intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to any number of items listed.

[0139] The flowcharts and block diagrams in the figures illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, segment or portion of an instruction, which includes one or more executable instructions for implementing (multiple) specified logical functions. In some alternative implementations, the functions indicated in the box may not occur in the order indicated in the figure. For example, the two boxes shown in succession can actually be completed as a step, concurrently, substantially concurrently, in a manner that overlaps in part or all of the time, or these boxes can sometimes be executed in the opposite order, depending on the functions involved. It will also be noted that each box in the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0140] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or a combination thereof that collaborate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In some embodiments, the hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.), which may be on-board or off-board, or may be dedicated for use by the hardware processor subsystem.

[0141] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0142] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0143] These and other variations of these and of hardware processor subsystems are according to embodiments of the present invention.

[0144] Having described preferred embodiments of video synthesis (which are intended to be illustrative and not limiting), it should be noted that modifications and variations can be made by those skilled in the art in light of the above teachings. Therefore, it should be understood that changes can be made in the specific embodiments disclosed that are within the scope of the invention as outlined by the appended claims. Having thus described the invention in its various aspects, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.

Claims

1. A method for implementing video frame synthesis using a tensor neural network, comprising: receiving input video data including one or more missing frames; Converting the input video data into an input tensor; generating output video data by using a transform-based tensor neural network (TTNet) based on the input tensor for tensor completion, the output video data comprising one or more synthesized frames corresponding to the one or more missing frames, the TTNet comprising a plurality of stages implementing a tensor iterative shrinkage thresholding algorithm (ISTA); as well as A loss function is obtained based on the output video data.

2. The method of claim 1 , wherein generating the output video data further comprises: For a given stage of the plurality of stages of the TTNet, transform an intermediate synthesis result in an original domain into a transformed intermediate synthesis result in a transformed domain, wherein the original domain is a domain where the input tensor is located; applying soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transformed domain; as well as The composite video data in the transformed domain is transformed back to the original domain using an inverse transform.

3. The method according to claim 2, further comprising: Prior to transforming the intermediate result in the original domain, the intermediate result in the original domain is updated, wherein the intermediate synthesis result is defined in part based on the video tensor and the observation tensor received by the given stage of the plurality of stages.

4. The method according to claim 2, wherein transforming the intermediate synthesis result in the original domain into the transformed intermediate synthesis result further comprises: A first convolution is applied to the intermediate synthesis result, an activation function is applied to the output of the first convolutional network, and a second convolution is applied to the output of the activation function to generate the transformation.

5. The method according to claim 4, wherein the first convolution and the second convolution comprise two-dimensional (2D) multi-channel convolutions with different kernels, and the activation function is a rectified linear unit (ReLU).

6. The method of claim 2, wherein applying soft thresholding based on the transformed intermediate synthesis result to generate the synthesized video data in the transformed domain further comprises: Applying a plurality of soft thresholding operations in parallel to each front slice of the transformed intermediate synthesis result and stacking outputs of the plurality of soft thresholding operations to generate the synthesized video data in the transformed domain, wherein the front slice is part of the input tensor.

7. The method of claim 2, wherein transforming the composite video data in the transformed domain back to the original domain using an inverse transform further comprises: A first convolution is applied to the synthesized video data in the transform domain, an activation function is applied to an output of the first convolution, and a second convolution is applied to an output of the activation function.

8. The method of claim 7, wherein the first convolution and the second convolution comprise 2D multi-channel convolutions with different kernels, and the activation function is ReLU.

9. The method of claim 1, wherein the loss function comprises a first portion that evaluates the accuracy of the one or more synthesized frames, and a second portion that imposes sparsity.

10. A computer program product comprising a computer-readable program for implementing video frame synthesis using a tensor neural network, wherein the computer program product, when executed on a computer, causes the computer to perform a method comprising: receiving input video data including one or more missing frames; Converting the input video data into an input tensor; generating output video data by using a transform-based tensor neural network (TTNet) based on the input tensor for tensor completion, the output video data comprising one or more synthesized frames corresponding to the one or more missing frames, the TTNet comprising a plurality of stages implementing a tensor iterative shrinkage thresholding algorithm (ISTA); as well as A loss function is obtained based on the output video data.

11. The computer program product of claim 10, wherein generating the output video data further comprises: For a given stage of the plurality of stages of the TTNet, transform an intermediate synthesis result in an original domain into a transformed intermediate synthesis result in a transformed domain, wherein the original domain is a domain where the input tensor is located; applying soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transformed domain; as well as The composite video data in the transformed domain is transformed back to the original domain using an inverse transform.

12. The computer program product of claim 11, wherein the method further comprises: The intermediate result in the original domain is updated before transforming the intermediate result into the original domain, and wherein the intermediate synthesis result is defined in part based on the video tensor and the observation tensor received by the given stage of the plurality of stages.

13. The computer program product of claim 11 , wherein: Transforming the intermediate synthesis result in the original domain into the transformed intermediate synthesis result further comprises: applying a first convolution to the intermediate synthesis result, applying a first rectified linear unit (ReLU) to an output of the first convolution, and applying a second convolution to an output of the first ReLU to generate the transformation, wherein the first convolution and the second convolution comprise two-dimensional (2D) multi-channel convolutions with different kernels; Transforming the composite video data in the transformed domain back to the original domain using an inverse transform further comprises: applying a third convolution to the composite video data in the transformed domain, applying a second ReLU to an output of the third convolution, and applying a fourth convolution to an output of the second ReLU; and The first convolution, the second convolution, the third convolution, and the fourth convolution include 2D multi-channel convolutions with different kernels.

14. The computer program product of claim 11 , wherein applying soft thresholding based on the transformed intermediate synthesis result to generate the synthesized video data in the transformed domain further comprises: Applying a plurality of soft thresholding operations in parallel to each front slice of the transformed intermediate synthesis result and stacking outputs of the plurality of soft thresholding operations to generate the synthesized video data in the transformed domain, wherein the front slice is part of the input tensor.

15. The computer program product of claim 11, wherein the loss function comprises a first portion that evaluates accuracy of the one or more synthesized frames, and a second portion that imposes sparsity.

16. A system for implementing video frame synthesis using a tensor neural network, comprising: a memory configured to store program code; as well as at least one processor device operatively coupled to the memory and configured to execute program code stored on the memory to: receiving input video data including one or more missing frames; Converting the input video data into an input tensor; generating output video data by using a transform-based tensor neural network (TTNet) based on the input tensor for tensor completion, the output video data comprising one or more synthesized frames corresponding to the one or more missing frames, the TTNet comprising a plurality of stages implementing a tensor iterative shrinkage thresholding algorithm (ISTA); as well as A loss function is obtained based on the output video data.

17. The system of claim 16, wherein the at least one processor device is further configured to generate the output video data by: For a given stage of the plurality of stages of the TTNet, transform an intermediate synthesis result in an original domain into a transformed intermediate synthesis result in a transformed domain, wherein the original domain is a domain where the input tensor is located; applying soft thresholding based on the transformed intermediate synthesis result to generate synthesized video data in the transformed domain; as well as The composite video data in the transformed domain is transformed back to the original domain using an inverse transform.

18. The system of claim 17, wherein: The at least one processor device is further configured to transform the intermediate synthesis result in the original domain into the transformed intermediate synthesis result by applying a first convolution to the intermediate synthesis result, applying a first rectified linear unit (ReLU) to an output of the first convolution, and applying a second convolution to an output of the first ReLU to generate the transformation; The at least one processor device is further configured to transform the composite video data in the transform domain back to the original domain using an inverse transform by applying a third convolution to the composite video data in the transform domain, applying a second ReLU to an output of the third convolution, and applying a fourth convolution to an output of the second ReLU; and The first convolution, the second convolution, the third convolution, and the fourth convolution include 2D multi-channel convolutions with different kernels.

19. The system of claim 17 , wherein the at least one processor device is further configured to apply soft thresholding based on the transform intermediate synthesis result to generate the synthetic video data in the transform domain by applying multiple soft thresholding operations in parallel to each front slice of the transform intermediate synthesis result, and stacking outputs of the multiple soft thresholding operations to generate the synthetic video data in the transform domain, wherein the front slice is part of the input tensor.

20. The system of claim 16, wherein the loss function comprises a first portion that evaluates the accuracy of the one or more synthesized frames, and a second portion that imposes sparsity.

Images