Method and electronic device for synthesizing video
The diffusion-based frame-to-video model with recursive sampling and explicit motion control addresses the inefficiencies of existing video synthesis methods by enabling high-quality video generation at higher resolutions with reduced resource requirements, achieving improved temporal coherence and efficiency.
Patent Information
- Application Number
- PCT/KR2024/005723
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-12
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-28
AI Technical Summary
Current video synthesis technologies face challenges in producing high-quality videos from a single frame due to the need for large-scale video datasets and substantial computational resources, making them expensive and impractical for widespread use.
A diffusion-based frame-to-video model with a recursive frame sampling scheme and explicit motion control using a shared motion latent code, allowing for efficient training on lower resolution videos and inference at higher resolutions without significant quality reduction.
The proposed solution enables the synthesis of high-quality videos at higher resolutions than current state-of-the-art techniques with a slight trade-off in quality metrics, demonstrating temporal coherence and efficiency in resource usage.
Smart Images

Figure KR2024005723_28112024_PF_FP_ABST
Abstract
Description
METHOD AND ELECTRONIC DEVICE FOR SYNTHESIZING VIDEO
[0001] The present application relates to autoregressive synthesis of high-quality videos from a single frame (image) using fully-convolutional diffusion model.
[0002] Video synthesis, a critical research area in computer vision and graphics, has widespread potential applications from personalized content creation to Computer-Generated Imagery (CGI) effects. Although recent advancements have improved the quality of image synthesis, high-resolution image-to-video animation remains a formidable challenge. The primary reason is that existing models necessitate large-scale video datasets and substantial computational resources, making them expensive to train and limiting their practicality.
[0003] To address challenges inherent to prior art solutions the technical solutions proposed and disclosed herein use a diffusion-based frame-to-next-frame model that in fact is a diffusion-based frame-to-video model with recursive frame sampling scheme and explicit motion control of the videos being synthesized via shared motion latent code. Advantageously, the technical solutions proposed and disclosed herein thus allow to maintain temporal coherence of the videos being synthesized. Moreover, application of the diffusion-based frame-to-video model with the recursive frame sampling scheme and explicit motion control makes it possible to train the model on lower resolution videos, using less Video Random Access Memory (VRAM) and then, when trained, use it for inference in higher resolution (e.g. up to 2048 Х 1280) without significant reduction in image quality.
[0004] According to an embodiment of the disclosure, a method for synthesizing a video is provided. The method may include obtaining a shared motion latent code associated with a motion between frames of the video by inputting an input frame to a first encoder. The method may include obtaining a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame. The method may include predicting, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation.
[0005] According to an embodiment of the disclosure, an electronic device synthesizing a video is provided. The electronic device comprises a memory storing computer-executable program and at least one processor. The at least one processor execute the program or at least one instruction stored in the memory to cause the electronic device to obtain a shared motion latent code associated with a motion between frames of the video by inputting an input frame to a first encoder. The at least one processor execute the program or at least one instruction stored in the memory to cause the electronic device to obtain a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame. The at least one processor execute the program or at least one instruction stored in the memory to cause the electronic device to predict, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation.
[0006] According to an embodiment of the disclosure, a computer-readable medium storing computer-executable instructions which when executed by an electronic device cause the electronic device to perform the method is provided. The method may include obtaining a shared motion latent code associated with a motion between frames of the video by inputting an input frame to a first encoder. The method may include obtaining a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame. The method may include predicting, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation.
[0007] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
[0008] Figure 1 schematically represents the architecture of the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0009] Figure 2 schematically represents the structure of the trained neural network comprised in the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0010] Figure 3 schematically represents the motion latent code sampling process performed by the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0011] Figure 4 illustrates examples of frames synthesized in an autoregressive way starting from an input frame with the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0012] Figure 5 illustrates a method for synthesizing vidieo according to embodiments of the present disclosure.
[0013] It should be understood at the outset that although various embodiments of the present disclosure are illustrated below, the present disclosure may be implemented using any number of techniques, whether currently known or in existence. The present disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary design and implementation illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.
[0014] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flowcharts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
[0015] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces
[0016] The term "some" as used herein is defined as "none, or one, or more than one, or all." Accordingly, the terms "none," "one," "more than one," "more than one, but not all" or "all" would all fall under the definition of "some." The term "some embodiments" may refer to no embodiments or one embodiment or several embodiments or all embodiments. Accordingly, the term "some embodiments" is defined as meaning "no embodiment, or one embodiment, or more than one embodiment, or all embodiments."
[0017] The terminology and structure employed herein are for describing, teaching, and illuminating some embodiments and their specific features and elements and do not limit, restrict, or reduce the spirit and scope of the claims or their equivalents.
[0018] More specifically, any terms used herein such as but not limited to "includes," "comprises," "has," "consists," and grammatical variants thereof do not specify an exact limitation or restriction and certainly do not exclude the possible addition of one or more features or elements, unless otherwise stated, and must not be taken to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated.
[0019] Whether or not a certain feature or element was limited to being used only once, either way, it may still be referred to as "one or more features" or "one or more elements" or "at least one feature" or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" feature or element do not preclude there being none of that feature or element unless otherwise stated. Thus, at least one of A, B or C may be referred to as "only a", "only b", "only c", "both a and b", "both a and c", “both b and c", "all of a, b, and c", or variations thereof.
[0020] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.
[0021] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.
[0022] Unless otherwise defined, all terms, and especially any technical and / or scientific terms, used herein may be taken to have the same meaning as commonly understood by one having ordinary skill in the art.
[0023] "Diffusion model" may refers to a neural network model including a class of generative models used for generating complex data distributions and generate new data samples.
[0024] "Attention" refers to a mechanism or technique that helps neural network model focus on specific parts of input data.
[0025] "Conditioning motion" refers to the process of taking into account the motion information from the input frames to make predictions about the motion and appearance of the next frame in diffusion models for predicting the next frame in a video sequence.
[0026] "A U-Net" may refers to a neural network model including an encoder and a decoder, with a bottleneck or bridge in between.
[0027] "A shared motion latent code" may refers to a data representing encoded motion patterns among objects or entities in video frames.
[0028] "A latent representation" may refers to a data representing an encoded frame for reducing spatial resolution of the frame.
[0029] "A pre-trained Variational Autoencoder (VAE)" is a neural network model that has been trained on a large dataset before being fine-tuned or adapted for a specific task or application.
[0030] "Embedding" may refers to the process of mapping data, often in the form of discrete or categorical variables, into a continuous, lower-dimensional space.
[0031] "A bank" may refer to a component or layer in the neural network model architecture that is responsible for handling embeddings. In an embodiment, the bank of learned motion embeddings is stored in motion embedding layer.
[0032] The technical solutions proposed and disclosed herein perform video synthesis task in an autoregressive way by conditioning the synthesis of a next frame by a input frame and, additionally, by shared motion latent code determined once for an input frame and kept constant for all synthesized frames. The shared motion latent code is a trainable vector that learns the motion dynamics of a video. The architecture of the diffusion-based frame-to-next-frame model synthesizing the frames requires only the current video frame to synthesize the next one, and thus ensures data-efficient and computationally-efficient training at high resolutions. Additionally, by conditioning all synthesized frames on a shared motion latent code, the diffusion-based frame-to-next-frame model achieves temporal consistency without memory-intensive temporal attention or temporal convolution layers.
[0033] The effectiveness of the proposed diffusion-based frame-to-next-frame model was acknowledged experimentally on well-known video synthesis datasets, such as SkyTimeLapse and DeepLandscape, using Frechet Video Distance (FVD) and Frechet Inception Distance (FID) as the main evaluation metrics. Experimental data obtained are given in the end of this detailed specification. By being trained on videos having the resolution 1280Х720 the proposed diffusion-based frame-to-next-frame model is able to produce videos in up to 2 times higher resolution than current state-of-the-art techniques obtaining only a slight trade-off in FVD.
[0034] The main contributions of the present patent application over the prior art are the technical solutions proposed and disclosed herein, which are devised with the understanding that complex temporal coherence mechanisms such as temporal attention and temporal convolution layers are unnecessary for smaller video datasets and that lighter models with a recursive sampling scheme can compete with the more powerful models on smaller training datasets. Thus, the diffusion-based frame-to-next-frame model used in the technical solutions proposed and disclosed herein represents a new class of video-diffusion models that consists of a Latent Diffusion Model (LDM) that is conditioned by a input frame of a video being synthesized and a shared latent motion code that is jointly learned for each video.
[0035] Diffusion models are a class of models often used in generative image modeling. According to one existing formulation, Denoising Diffusion Probabilistic Models (DDPMs) are latent variable models of the form , where are latent variables of the same dimensionality as the data . Thediffusion process, or theforward process, is a Markov chain with Gaussian transitions. It gradually adds Gaussian noise to the data according to a variance schedule :
[0036] (1)
[0037] Sampling at an arbitrary timestep during the forward process can also be done in closed form:
[0038] (2)
[0039] where and .
[0040] Thegenerative process, or thereverse process, is the joint distribution also defined as a Markov chain with learned Gaussian transitions starting at :
[0041] (3)
[0042] Learning the reverse process is equal to learning to denoise into an estimate for all timesteps . Thus, the proposed diffusion-based frame-to-next-frame model is optimized (trained) by minimizing a noise-prediction mean squared error loss (also referred to as L2 loss):
[0043] (4)
[0044] over times uniformly sampled from , where = and is the added noise predicted by the model estimation.
[0045] In an embodiment, a latent representation is obtained by inputting the input frame to a second encoder 102 to reduce a resolution of the input frame. In an embodiment, the second encoder includes an encoder of a pre-trained Variational Autoencoder (VAE). To generate higher-resolution videos, adopted herein is the LDM approach. Utilizing an second encoder 102 of a pre-trained VAE from Stable Diffusion all the data processed (at both training and inference stages) by the diffusion-based frame-to-next-frame model are encoded to reduce the spatial resolution. In the particular non-limiting implementation of the encoding the spatial resolution is reduced by Х8. Using the learned latent space, latent representations of the frames synthesized by the diffusion-based frame-to-next-frame model are then decodable by a decoder of the same VAE back to the pixel space (i.e. into actually synthesized frames). This feature provides the possibility of implementing the proposed technical solutions on devices with limited processor and memory resources.
[0046] Beside unconditional generation, diffusion models are capable of modeling conditional distributions and solving such tasks as class-conditional image synthesis, text-to-image or text-to-video generation, image inpainting and various other image-to-image translation tasks. To synthesize video with consecutive frames the diffusion-based frame-to-next-frame model disclosed herein is conditioned on a input frame and, additionally, on a video-specific motion latent code. More particularly, a conditioning frame is concatenated with the latent representation and provided them jointly to the diffusion-based frame-to-next-frame model trained to synthesize a latent representation decodable by the decoder of the VAE into a next frame.
[0047] In an embodiment, Figure 1 schematically represents the architecture of the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0048] Now with reference to Figure 1 the architecture of the diffusion-based frame-to-next-frame model according to a non-limiting embodiment of the present disclosure will be described in detail. The diffusion-based frame-to-next-frame model comprises the second encoder 102 and decoder 108 of the pretrained VAE, a trained neural network model, and shared motion latent code obtained by motion embedding 104 and style modulation. In an embodiment, the neural network model includes U-Net-based diffusion predictor. The architecture of the proposed U-Net-based diffusion predictor will be described in detail with reference to Figure 2.
[0049] The diffusion-based frame-to-next-frame model does not rely on temporal attention, temporal convolution, or any other operation that explicitly propagate information across video frames. To maintain temporal coherence, the video is synthesized in an autoregressive way, frame-by-frame, by predicting the next frame from the input frame and shared motion latent code that is the same for all frames in a single video. At inference, is the initial frame input to the diffusion-based frame-to-next-frame model. To synthesize high-resolution videos, the diffusion-based frame-to-next-frame model is configured to operate in a low-dimensional latent space of the pre-trained VAE being the perceptual compression model. This VAE consists of the second encoder 102 and a decoder 108. The VAE is pretrained in advance on the task of random frames autoencoding, i.e. an input training frame passes through the second encoder 102, becomes a latent representation, and returns back its original pixel space after passing through the decoder 108. The second encoder 102 and decoder 108 may be trained, but without the limitation, with a loss function computed between the original ground-truth original frame and a reconstructed frame obtained by the decoder 108. Alternatively, the VAE including the second encoder 102 and the decoder 108 may be trained in an end-to-end manner with the U-Net-based diffusion predictor comprised in the diffusion-based frame-to-next-frame model to compress input frames and reconstruct synthesized frames. To train the diffusion-based frame-to-next-frame model pairs of frames are randomly sampled at each pass of the model from a training video of the training dataset, where denotes the video time delta between the sampled frames. In an embodiment, the pairs of frames may include input frame and next frame. Training dataset may comprise any available videos, e.g. videos of people, clouds, waves, tree foliage, etc.
[0050] It is known that at least certain adjacent frames (e.g. frames of substantially a same scene) in videos are often similar to one another and will accumulate change as they get further from the starting (initial) frame of the adjacent frames. Considering this, next (synthesized) frame latent representation is modeled by the technical solutions as the sum of the input frame latent representation and the residual predicted by the U-Net-based diffusion predictor (named as "U-Net" in Figure 1) comprised in the diffusion-based frame-to-next-frame model:
[0051] , (5)
[0052] where denotes the video time delta between input frame and predicted frame . The residual is predicted by the U-Net-based diffusion predictor (LDM) as follows:
[0053]
[0054] where DP is the diffusion process 106 described above with reference to mathematical expressions (1) to (4) above and implemented by the U-Net-based diffusion predictor denotes obtained-by-training weights of is a starting noise sampled from normal distribution is conditioning input frame latent representation, and is the shared motion latent code. In an embodiment, input frame the latent representation may be a first latent representation and the next frame latent representation may be a second latent representation.
[0055] While the nature of motion found in videos can vary significantly between various domains, e.g., facial animation and landscape animation, within the scope of this patent application, it is assumed that the motion pattern of the objects in videos from a single domain, i.e., people, clouds, waves, tree foliage, etc. is describable by a shared motion latent code , which is jointly learned for each video in the training dataset and applied in with style modulation. In a non-limiting implementation the shared motion latent code may be in the form of a vector of size . Experimental data given below in the end of the present specification are obtained with being set to 16. However, depending on different circumstances (e.g. depending on currently available memory or processor resources) the size of the vector of the low-dimensional motion latent code may be set to higher or lower value than 16. Utilizing the shared motion latent code is essential for synthesizing videos of favorable quality, in which motion type, speed, and direction are maintained (due to conditioning of inference by the shared motion latent code ) consistent throughout synthesized frames.
[0056] Consistent with the above description an overview of the architecture of the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure is schematically illustrated in Figure 1. As follows from Figure 1 the pipeline comprises at least the following operations of: projecting the input frame from pixel space to latent vector space by passing the input frame through the second encoder 102 of the VAE, as the result of the operation the first latent representation of the input frame is obtained; then concatenating the initial noised residual (being the noise sampled from the normal distribution) with the first latent representation of the input frame and passing the result of the concatenation through the U-Net-based diffusion predictor while conditioning motion between the frames of the video being synthesized with the shared motion latent code (depicted in Figure 1 as "Style Modulation" block), as the result of the operation the denoised residual is predicted by the U-Net-based diffusion predictor; then the predicted denoised residual is added to the first latent representation of the input frame to model the second latent representation of the next frame ; and finally, the second latent representation is projected back to the pixel space by the decoder 108 of the VAE to obtain the synthesized next frame .
[0057] A detailed but not limiting implementation of the architecture of the U-Net-based diffusion predictor comprised in the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure is illustrated in Figure 2 and described below in detail. It will be understood by a skilled person in the art that the number the number of blocks and channels in the illustrated U-Net-based diffusion predictor may vary to lighten the model or make it more powerful for learning a more extensive dataset.
[0058] In an embodiment, Figure 2 schematically represents the structure of the trained neural network comprised in the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure. In an embodiment, the trained neural network includes the U-Net-based diffusion predictor.
[0059] The U-Net-based diffusion predictor comprises blocks of five types: (1) DoubleConv block 202, (2) DownBlock block 204, (3) DoubleConvCond block 206, (4) UpCondBlock block 208, (5) 64->4 block 210.
[0060] The first block of the U-Net-based diffusion predictor, which is namely the (1) DoubleConv block 202, operates on the latent representation of the input frame concatenated with the initial noised residual . The DoubleConv block 202 comprises a sub-block of 3Х3 2D convolution (Conv2d) with layer normalization (LayerNorm) followed by a sub-block of Gaussian Error Linear Unit (GELU) activation function with additional 3Х3 2D convolution (Conv2d) and layer normalization (LayerNorm). The input channel size of the DoubleConv block 202 is 8, the output channel size of the DoubleConv block 202 is 256.
[0061] The second block of the U-Net-based diffusion predictor, which is namely the (2) DownBlock block 204, operates on the output of the first block and additionally on a current diffusion timestep index which is drawn into the DownBlock block 204 via sine and / or cosine positional encoding. The DownBlock block 204 comprises a sub-block of Sigmoid Linear Unit (SiLU) activation function and a sub-block of max pooling (MaxPool) followed by x2 DoubleConv, which contents correspond to the contents of the DoubleConv block 202 described above. The resulted data of the sub-blocks are summed up and passed to the next block of the U-Net-based diffusion predictor. The input channel size of the second DownBlock block 204 is 256, the output channel size of the second DownBlock block 204 is 512.
[0062] The third and fourth blocks of the U-Net-based diffusion predictor are each the same as the above described DownBlock block 204. Thus, the description of contents of the blocks is omitted herein. The input channel size of the third DownBlock block 204 is 512, the output channel size of the third DownBlock block 204 is 1024. The input channel size of the fourth DownBlock block 204 is thus 1024, the output channel size of the fourth DownBlock block 204 is 2048. The second, third, and fourth DownBlock block 204s are further each connected via a respective skip connection respectively to the eighth, ninth and tenth UpCondBlock block 208s described below.
[0063] The fifth block of the U-Net-based diffusion predictor, which is namely the (3) DoubleConvCond block 206, operates on the output of the fourth block and additionally on the motion latent code . The DoubleConvCond comprises the same contents as the DoubleConv block 202 described above and additionally a sub-block of Style (including Motion) Modulation (SM). The description of contents of the sub-blocks of the DoubleConv block 202 is omitted herein. The sub-block of SM is used in the DoubleConvCond block 206 to concatenate the conditioning motion latent code to the results of the sub-blocks of the DoubleConv block 202. The input channel size of the fifth DoubleConvCond block 206 is 2048, the output channel size of the fifth DoubleConvCond block 206 is 2048.
[0064] The sixth and seventh blocks of the U-Net-based diffusion predictor are each the same as the above described DoubleConvCond block 206. Thus, the description of contents of the blocks is omitted herein. The input channel size of the sixth DoubleConvCond block 206 is 2048, the output channel size of the sixth DoubleConvCond block 206 is 2048. The input channel size of the seventh DoubleConvCond block 206 is 2048, the output channel size of the seventh DoubleConvCond block 206 is 2048.
[0065] The eighth block of the U-Net-based diffusion predictor, which is namely the (4) UpCondBlock block 208, operates on the output of the seventh block and additionally on the motion latent code and the current diffusion timestep index drawn into the UpCondBlock block 208 via sine and / or cosine positional encoding. The UpCondBlock block 208 comprises a sub-block of Sigmoid Linear Unit (SiLU) activation function and a sub-block of DoubleConvCond and DoubleConv, which contents respectively correspond to the contents of the DoubleConvCond block 206 and DoubleConv block 202 described above. The resulted data of the sub-blocks are summed up and passed to the next block of the U-Net-based diffusion predictor. The input channel size of the eighth UpCondBlock block 208 is 4096, the output channel size of the eighth UpCondBlock block 208 is 512.
[0066] The ninth and tenth blocks of the U-Net-based diffusion predictor are each the same as the above described UpCondBlock block 208. Thus, the description of contents of the blocks is omitted herein. The input channel size of the ninth UpCondBlock block 208 is 1024, the output channel size of the ninth UpCondBlock block 208 is 256. The input channel size of the tenth UpCondBlock block 208 is 512, the output channel size of the tenth UpCondBlock block 208 is 64.
[0067] The last block of the U-Net-based diffusion predictor, which is namely the 64->4 block 210, is the fully connected block with 1Х1 2D convolution (Conv2d).
[0068] Training procedure applied to the diffusion-based frame-to-next-frame model is described in detail in the followingPseudocode 1:
[0069] Pseudocode 1: the diffusion-based frame-to-next-frame model training procedure (without loss of generality it is assumed here that batch size isequal to 1)
[0070] repeat
[0071] Sample: and
[0072] ,
[0073]
[0074]
[0075]
[0076]
[0077] Take gradient descent step on
[0078]
[0079] untilConverged (i.e. until the global optimum value of the loss function is obtained)
[0080] where is a hyperparameter of .
[0081] The diffusion-based frame-to-next-frame model may be trained not on entire frames but on crops of the frames. In a particular non-limiting implementation the diffusion-based frame-to-next-frame model is trained on 5 crops of each of one or more frames of one or more full-resolution training videos comprised in a training dataset: the center and four corners, to make the motion latent representations robust. These 5 crops share the same motion latent code. This helps to reduce the dependence on the spatial arrangement of moving pixels in the video. In addition, the motion diversity of the training dataset may optionally be increased through augmentations: horizontal flip and time inversion applied to training frames / crops. The result is a 4-fold increase in the number of unique motion latents. Once is trained, the bank of learned motion embeddings stored in motion embedding layer is obtained. The motion embedding layer is an array of vectors from which the desired embedding is obtained by the video index. Initially, for each video, the vector is initialized randomly. At each iteration of training, one vector is taken from the array by index, corresponding to the video whose frames participate in this iteration. Next, participates in the diffusion process and is updated by gradient descent like all other weights.
[0082] In an embodiment, Figure 3 schematically represents the motion latent code sampling process performed by the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0083] In a particular non-limiting embodiment of the shared motion latent code sampling scheme obtaining the shared motion latent code by a first encoder. In an embodiment, the first encoder 304 includes Contrastive Language-Image Pretraining (CLIP) encoder. In an embodiment, the shared motion latent code is used at inference of the diffusion-based frame-to-next-frame model for conditioning the dynamics of motion to be reflected in synthesized frames Contrastive Language-Image Pretraining (CLIP) approach is used as illustrated in Figure 3. However, the present disclosure should not be limited to the usage of CLIP, because to compare frame content it is possible to use various pre-trained image encoders. The shared motion latent code has to be selected properly to drive the direction and nature of the movement in the video being synthesized. A naive approach is to sample motion latent code from one or more motion latent codes of random videos, which are stored in the bank of learned motion embeddings 312. However, it was found that synthesizing an image while conditioning it with the motion latent code sampled from a drastically different video in terms of the content, i.e. in this case motion latent code will not be the shared one, may lead to visual artifacts, unrealistic movements, and unnaturally looking objects. Frame embedding is a set of feature maps obtained by from a frame, and motion embeddings 312 is a vector obtained from an array of learnable vectors by video index.
[0084] In Figure 3 depicted is the shared motion latent code sampling scheme, according to which the motion latent code of the videos with the closest CLIP scores is reused as the shared motion latent code . This results in smoother and more coherent video synthesis / animations. As illustrated in Figure 3 to obtain the shared motion latent code for a frame of a video being synthesized it is proposed first to obtain CLIP embedding for the frame by passing the frame through an encoder of the trained CLIP neural network that is known from the prior art as the OpenAI product. Then one or more CLIP embeddings 310 nearest to the obtained CLIP embedding is / are searched using k-Nearest Neighbors (kNN) 308 on the database of CLIP embeddings 306 obtained in advance for one or more frames of one or more training videos comprised in the training dataset used for training the diffusion-based frame-to-next-frame model. kNN 308 was chosen as the primary algorithm used herein, because it is the easiest to implement. However, the present disclosure should not be limited to the usage of kNN 308, because any available more complex algorithms may be used in the present disclosure as well. CLIP embeddings 310 may be obtained and stored in the database of CLIP embeddings 306 in advance to not re-calculate the CLIP embeddings 310 each inference time. Finally, the shared motion latent code is obtained from the bank of learned motion embeddings 312 corresponding to one or more frames of the one or more training videos that were used to obtain the one or more found CLIP embeddings 310. In a particular non-limiting embodiment the shared motion latent code of one or more starting frames of the one or more training videos that were used to obtain the one or more nearest CLIP embeddings 310 is reused as the shared motion latent code . A starting frame may be the input frame 302. In an embodiment, averaging may be applied to shared motion latent codes of the one or more training videos that were used to obtain the one or more nearest CLIP embeddings 310 and the averaged motion latent code may be used as the the shared motion latent code .
[0085] Inference procedure carried out by the trained diffusion-based frame-to-next-frame model is described in detail in the followingPseudocode 2:
[0086] Pseudocode 2: Inference procedure by the trained diffusion-based frame-to-next-frame model
[0087] Input - starting frame.
[0088] Select shared motion latent code from the bank of learned motion embeddings 312 (as described with reference toFigure 3above)
[0089]
[0090] for do
[0091]
[0092] for do
[0093]
[0094]
[0095] end for
[0096]
[0097]
[0098] end for
[0099] Return:
[0100] Thus, is used at inference to generate a video sequence given the starting (conditioning) frame in an autoregressive way. Since the diffusion-based frame-to-next-frame model is fully convolutional, advantageously it is possible to use at inference the starting frame of higher resolution than that of frames used at training of said model.
[0101] The method of synthesizing a video as described above may be performed by an electronic device (not shown). Such a device may be, but not limited to, a smartphone, a tablet, a notebook, PC, AR / VR headset and so on. The electronic device may at least comprise a processor and a memory storing computer-executable instructions which when executed by the processor cause the device to perform the method according to the first aspect or according to any development of the first aspect. The memory may further directly store weights and offsets of the diffusion-based frame-to-next-frame model including the U-Net-based diffusion predictor and VAE. In addition, the memory may further directly store weights and offsets of CLIP neural network.
[0102] Alternatively, the electronic device equipped with a communication unit may send a request to one or more servers executing the diffusion-based frame-to-next-frame model including the U-Net-based diffusion predictor and VAE and the CLIP neural network to receive in response to the request one or more, or all externally synthesized frames of video sequence(s). In this case, the technical solutions proposed and described herein may be implemented on such one or more computer-implemented servers. Moreover, training of the diffusion-based frame-to-next-frame model including the U-Net-based diffusion predictor and VAE and, if necessary, of the CLIP-based model may be online (i.e. be executed on the device itself) or offline (i.e. be executed by one or more external servers).
[0103] The processor may be of any type, e.g., it may include, but without any limitation, one or more of: a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Processor (AP), a Graphics-Processing Unit (GPU), a Vision Processing Unit (VPU), a dedicated AI processor such a Neural Processing Unit (NPU) and so on. The processor may be implemented as System-on-Chip (SOC), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other Programmable Logic Device (PLD), discrete logic element, transistor logic, discrete hardware components, or any combination thereof.
[0104] The memory may be of any type, e.g., it may include, but without any limitation, one or more of: Random Access Memory (RAM), Video Random Access Memory (VRAM), Dynamic RAM (DRAM), Static RAM (SRAM), Double Data Rate SDRAM (DDR SRAM), Double Data Rate 4 Synchronous Dynamic RAM (DDR4 RAM), Rambus Dynamic RAM (DRDRAM), Read-Only Memory (ROM), Programmable ROM (PROM), Erasable PROM (EPROM), Electrically erasable PROM (EEPROM), virtual memory. The memory may be implemented as Solid-State Drive (SSD), Hard Disk Drive (HDD), USB Flash Drive and so on.
[0105] The electronic device may operate on any operating system (e.g. Android, iOS, Harmony OS, Windows, Linux etc.) and may include any other necessary software, firmware, and / or hardware (e.g. a communication unit, I / O interface, a camera (e.g. to capture the staring frame ), a power supply and so on). Non-limiting examples of computing device 50 include a smartphone, a smartwatch, a tablet, a hearing aid device, a computer, a notebook, AR / VR headset and so on.
[0106] The present application also provides a computer-readable (non-transitory) medium storing computer-executable instructions which when executed by a device cause the device to perform the method (or function as the device performing the method) according to the first aspect or according to any development of the first aspect. Any types of media or storage devices may be used as the computer-readable medium.
[0107] Experimental data and other non-limiting features of the proposed technical solutions. Provided below are quantitative and qualitative experiment results. The flagship model was trained on the DeepLandscape dataset. The dataset consists of 999 training landscape videos in 1280Х720 resolution in a training split and 57 testing videos. For comparison with quantitative existing solutions for image-to-video generation, the diffusion-based frame-to-next-frame model proposed and disclosed herein was trained and evaluated on the SkyTimelapse dataset of clips of different length containing dynamic sky scenes. It contains 35392 training video clips and 2815 testing video clips in 640Х360 resolution. The scenes include different daytime and weather conditions. Both diffusion-based frame-to-next-frame models were trained on Nvidia A100 80GB GPU (Х4 GPUs for DeepLandscape dataset) with a batch size of 256 and AdamW optimizer with the learning rate set to . The SkyTimelapse models were trained on 256 Х 256 crops, and DeepLandscape models - on 512 Х 512 crops. In all experiments, the video time delta was set to 1 and the model was trained for 150K epochs, where one epoch is one full pass through a dataset.
[0108] Quantitative results are given in below Table 1: Quantitativeresults of image-to-video generation quality and diversity on SkyTimelapse dataset:
[0109]
[0110] Conditional Video Generation. The quality of synthesized frames was evaluated via LPIPS and FID. For video quality assessment, reported are FVD and DTFVD, which is helpful to measure other types of dynamics not captured by FVD. Further, the diversity of generated videos was measured by DIV VGG and DIV I3D. Since the diffusion-based frame-to-next-frame model is able to generate several different videos from a given initial frame, both metrics measure their average mutual distance in the feature space of a VGG-16 and I3D networks pre-trained on ImageNet and Kinetics respectively. To compare fairly with competitors, metrics calculation methods were adopted from Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G. Derpanis, and Bjorn Ommer "Stochastic image-to-video synthesis using cinns" in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Evaluation was performed on the videos of 128Х128 resolution and 32 frames long.
[0111] It is seen from the results given in the above Table 1, that while the diffusion-based frame-to-next-frame model proposed and disclosed herein achieves comparable quality to that of the competitors, it generates more diverse videos and significantly outperforms the other models according to both diversity metrics. Moreover, unlike existing approaches, diffusion-based frame-to-next-frame model can produce videos in higher resolutions than the videos in the training set.
[0112] Unconditional Video Generation. Even though most of the video generation models solve unconditional video generation tasks, it is important to compare the results achieved by the diffusion-based frame-to-next-frame model with them. For that, in the experiments the first frame was generated with the StyleGAN-V generator and then animated with the diffusion-based frame-to-next-frame model proposed and disclosed herein. These experiments were conducted on the SkyTimelapse dataset as well. 162 generated videos with a resolution of 256Х256 and a length of 16 and 128 frames were assessed. Obtained quantative results of unconditional video generation quality on SkyTimelapse dataset are reported in the belowTable 2:
[0113]
[0114] The results given in the above Table 2 demonstrate that with the same initial frames, the diffusion-based frame-to-next-frame model generates better videos in terms of FVD than StyleGAN-V. Moreover, the advantage remains for videos of 128 frames, although the diffusion-based frame-to-next-frame model is autoregressive compared to the competitor.
[0115] In an embodiment, Figure 4 illustrates examples of frames synthesized in an autoregressive way starting from an input frame with the diffusion-based frame-to-next-frame model according to an embodiment of the present disclosure.
[0116] Illustrated on Figure 4 are examples for different frame sequences generated for a single input frame by the diffusion-based frame-to-next-frame model trained on DeepLandscape dataset in 1280Х704 resolution. Demonstrated are the 1st, 6th, 11th and 16th generated frames. Using different motion latent codes from the bank of learned motion embeddings 312, the direction and nature of the motion on the video are modifiable.
[0117] In favor of high-resolution video generation and due to memory limitations, self-attention layers originally used in DDPMs are eliminated from the proposed diffusion-based frame-to-next-frame model. This makes the proposed diffusion-based frame-to-next-frame model fully convolutional and enables generating videos of higher resolution than the ones the model was trained on. The experiments conducted by authors of the present disclosure reveal that this simplification does not bring any quality loss. Metrics also confirm that utilizing style modulation (with the shared motion latent code ) for conditioning is comparable to the Cross-Attention technique which is popular for diffusion models. However, advantageously, style modulation is more memory efficient than Cross-Attention and keeps the diffusion-based frame-to-next-frame model fully convolutional.
[0118] The diffusion-based frame-to-next-frame model is trained using a number of crops (e.g. 5 crops) of the video with shared motion latent code. This type of data augmentation regularizes the model and prevents dependence of learned motion latent codes on the spatial regions of the training frames. The model trained on one center crop generates separate frames of good quality, but video consistency suffers. This can be seen in the considerable FVD and diversity metrics increase. It is also shown that the kNN-based shared motion latent code sampling scheme increases video quality and diversity compared to the random sampling of the motion latent code.
[0119] Thus, disclosed in the present application is a new class of video-diffusion models that generated videos in 2 times higher resolution than current state-of-the-art approaches achieving comparable video quality on small-scale video datasets. It has been shown that recursive frame sampling combined with a shared style / motion conditioning vector leads to better temporal coherence in data and resource-limited settings compared to more complex known architectures. The proposed architecture of the diffusion-based frame-to-next-frame model with a limited number of parameters can achieve high visual quality at large resolutions without training on proprietary large-scale video datasets opens new possibilities in generative video research and development regarding even more efficient models that apply alternative techniques to the popular and computationally expensive temporal attention and temporal convolution. It has been also acknowledged herein that the simple content-matching scheme using a pre-trained image encoder significantly improves the visual appeal of synthesized videos by selecting the shared motion latent codes that produce smoother animations without over-reliance on the semantic layouts of specific videos.
[0120] The purpose of this disclosure is to address various technical problems, which are not restricted solely to the ones mentioned earlier. Any other technical problems not explicitly stated here will be readily understood by those skilled in the art from the following disclosure.
[0121] In an embodiment, Figure 5 illustrates a method 500 for synthesizing vidio according to embodiments of the present disclosure.
[0122] In an embodiment, at step S502, a method may include obtaining a shared motion latent code corresponding to a motion between frames of the video by inputting an input frame to a first encoder. At step S504, the method may include obtaining a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame. At step S506, the method may include predicting, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation
[0123] In an embodiment, the first encoder includes a Contrastive Language-Image Pretraining (CLIP) encoder; wherein the second encoder includes pre-trained Variational Autoencoder (VAE); wherein the trained neural network model includes a U-Net-based diffusion predictor.
[0124] In an embodiment, the method includes predicting, by using the trained neural network model, one more next frame from the previously predicted next frame based on the shared motion latent code and the first latent representation.
[0125] In an embodiment, the motion between the frames of the video being synthesized is based on the shared motion latent code by inputting the shared motion latent code to one or more layers of the trained neural network model.
[0126] In an embodiment, the method includes obtaining a Contrastive Language-Image Pretraining (CLIP) embedding for the input frame by inputting the input frame to the first. The method includes identifying one or more CLIP embeddings nearest to the obtained CLIP embedding using a k-Nearest Neighbors (kNN) and a distance metric for the kNN, based on a database of CLIP embeddings obtained for one or more frames of one or more training videos comprised in a training dataset used for training the neural network model. The method includes obtaining the shared motion latent code from a motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more CLIP embeddings.
[0127] In an embodiment, the distance metric for the kNN is computed as the distance of the CLIP embedding of a frame of a training video comprised in the training dataset and the CLIP embedding of the input frame.
[0128] In an embodiment, the method includes obtaining a pair of frames including the input frame and the next frame from a training video of a training dataset. The method includes obtaining the first latent representation and a second latent representation of the pair of frames by inputting the pair of frames to the second encoder. The method includes computing a residual by subtracting the first latent representation from the second latent representation. The method includes obtaining the shared motion latent code of the training video indicated by the index, from which the pair of frames are obtained, and storing the obtained shared motion latent code to the database of CLIP embeddings. The method includes obtaining a diffusion timestep index corresponding to the ordinal of the diffusion timesteps, wherein the diffusion timestep index is drawn into the neural network model based on sine or cosine function. The method includes obtaining a sampled noise tensor by sampling a noise tensor based on a normal distribution. The method includes obtaining a predicted noise tensor based on the shared motion latent code, the diffusion timestep index, the first latent representation, and the residual. The method includes updating, based on the sampled noise tensor, the predicted noise tensor, weights of the neural network model.
[0129] In an embodiment, the method includes obtaining the predicted noise tensor by inputting the shared motion latent code, the diffusion timestep index, the first latent representation, and the computed residual to the neural network model.
[0130] In an embodiment, the pair of frames obtained from the training video are frames located adjacently or near to each other in the sequence of frames of the training video. The method includes minimizing a loss function between the predicted noise tensor and the sampled noise tensor by using a gradient descent.
[0131] In an embodiment, the method includes obtaining crops of each of the pair of frames including a center and one or more of four corners of the pair of frames. The method includes obtaining the first latent representations and the second latent representation of the pair of frames by inputting the pair of frames to the second encoder in the form of the crops of the pair of frames.
[0132] In an embodiment, the method includes obtaining additional pairs of frames by performing at least one of horizontal flip or time inversion of the pair of frames from the training video of the training dataset. The method includes obtaining the CLIP embedding for the input frame. The method includes identifying one or more CLIP embeddings nearest to the obtained CLIP embedding using kNN, based on the database of the CLIP embeddings obtained previously for one or more training videos comprised in the training dataset used for training the neural network model. The method includes obtaining the shared motion latent code based on a bank of learned motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more CLIP embeddings. The method includes obtaining the first latent representation by inputting the input frame to the second encoder. The method for each frame in the video being synthesized by using the trained neural network model starting with the next frame includes: obtaining an initial noised residual, obtaining a denoised residual by performing steps of a diffusion by the trained neural network model, wherein for each the steps of the diffusion include: subtracting from a noised residual an output value of the trained model, obtaining the second latent representation of a next frame by adding the denoised residual to the first latent representation of the input frame, obtaining the next frame by inputting the second latent representation to a decoder corresponding to the second encoder and obtaining the synthesized video by concatenating all obtained frames.
[0133] In an embodiment, a method of synthesizing a video (a sequence of frames) in an autoregressive way includes: predicting with the use of a trained diffusion-based frame-to-next-frame model a next frame from an input frame while conditioning motion between the frames of the video being synthesized with a shared motion latent code .
[0134] In an embodiment, the method further includes predicting with the use of the trained diffusion-based frame-to-next-frame model one more next frame from the previously predicted frame while still conditioning motion between the frames of the video being synthesized with said shared motion latent code .
[0135] In an embodiment, motion between the frames of the video being synthesized is conditioned (explicitly controlled) with the shared motion latent code by inputting said shared motion latent code to one or more layers of a U-Net-based diffusion predictor of the diffusion-based frame-to-next-frame model.
[0136] In an embodiment, the shared motion latent code is obtained by performing the steps of: obtaining Contrastive Language-Image Pretraining (CLIP) embedding for the input frame by passing the input frame through an encoder of the trained CLIP neural network; searching for one or more CLIP embeddings nearest to the obtained CLIP embedding using k-Nearest Neighbors (kNN), the search is carried out on a database of CLIP embeddings obtained for one or more frames of one or more training videos comprised in a training dataset used for training the diffusion-based frame-to-next-frame model; and sampling the shared motion latent code from a bank of learned motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more found CLIP embeddings.
[0137] In an embodiment, the distance metric for the kNN is computed as the distance from CLIP embedding of a frame of a training video comprised in the training dataset to the CLIP embedding of the input frame .
[0138] In an embodiment, the diffusion-based frame-to-next-frame model is trained by repeatedly performing the following steps until convergence: randomly sampling a pair of frames and from a training video of the training dataset, where denotes the video time delta between the frames; passing the frames and through an encoder of a pre-trained Variational Autoencoder (VAE) to obtain respective latent representations and of the frames; computing residual by subtracting the latent representation from the latent representation ; obtaining the shared motion latent code of the training video indicated by the index n, from which the frames and are sampled, and storing the obtained shared motion latent code to the database of learnable motion embeddings; randomly sampling the diffusion timestep index corresponding to the ordinal of the diffusion timesteps from 1 To , where is the total number of diffusion timesteps and the diffusion timestep index is drawn into the diffusion-based frame-to-next-frame model via sine and / or cosine positional encoding; randomly sampling a noise tensor from ; and updating weights of the U-Net-based diffusion predictor comprised in the diffusion-based frame-to-next-frame model by taking a gradient descent step on the L2 loss between sampled noise tensor and noise tensor predicted by the U-Net-based diffusion predictor conditioned on the sampled shared motion latent code , the diffusion timestep index , the latent representation , and computed residual .
[0139] In an embodiment, at diffusion timestep for frame the predicted noise tensor is obtained by passing through the U-Net-based diffusion predictor the sampled shared motion latent code , the diffusion timestep index , the latent representation , and computed residual noised with the randomly sampled noise tensor .
[0140] In an embodiment, the two frames sampled from the training video are frames located adjacently or near to each other in the sequence of frames of the training video, the diffusion-based frame-to-next-frame model conditioned on the sampled shared motion latent code , the diffusion timestep index , the latent representation , and computed residual is trained by minimizing the L2 loss between the predicted noise tensor and the sampled noise tensor .
[0141] In an embodiment, the two frames that are passed through the encoder of the pre-trained VAE to obtain respective latent representations and of the frames are passed in the form of crops of said frames, wherein the method further comprising the step of obtaining said crops from each of the frames, the crops of each of the frames include the center and one or more of four corners of the frame.
[0142] In an embodiment, the method further include one or more steps of obtaining additional pairs of frames to be used at the training stage by performing horizontal flip and / or time inversion of frame pairs randomly sampled from training videos comprised in the training dataset.
[0143] In an embodiment, predicting the next frame from the preceding frame with the use of the trained diffusion-based frame-to-next-frame model while conditioning motion between the frames of the video being synthesized with the shared motion latent code includes the steps of: obtaining CLIP embedding for the preceding frame ; searching for one or more CLIP embeddings nearest to the obtained CLIP embedding using kNN, the search is carried out on the database of CLIP embeddings obtained previously for one or more frames of one or more training videos comprised in the training dataset used for training the diffusion-based frame-to-next-frame model; sampling the shared motion latent code from a bank M of learned motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more found CLIP embeddings; passing the preceding frame through the encoder of the pre-trained VAE to obtain respective latent representation ; for each frame in the video being synthesized with the trained diffusion-based frame-to-next-frame model starting with the second frame: sampling the initial noised residual from ; obtaining the denoised residual by performing steps of diffusion by the U-Net-based diffusion predictor, wherein at step of diffusion: subtracting from the noised residual the output value of the trained U-Net-based diffusion predictor comprised in the diffusion-based frame-to-next-frame model , where is a hyperparameter of the trained U-Net-based diffusion predictor; obtaining the latent representation of a synthesized next frame by adding the denoised residual to the latent representation of the previous frame; obtaining the synthesized next frame by passing the latent representation of said frame through a decoder of the pre-trained VAE; and obtaining the synthesized video by concatenating all synthesized frames .
[0144] In all materials of the present application, the reference to an element in the singular does not exclude the presence of a plurality of such elements in an actual implementation of the disclosure, and, vice versa, the reference to an element in the plural does not exclude the presence of only one such element in an actual implementation of the disclosure. Any particular value or range of values indicated above is not to be interpreted in a limiting sense, but instead such a particular value or range of values are to be considered as representing a midpoint of a certain greater range, up to approximately 50% at both sides of the specified value or specifically indicated smaller range.
[0145] While this disclosure has been shown and described with reference to its various embodiments, those skilled in the art should understand that various changes in form and content can be made without departing from the spirit and scope of this disclosure, as defined by the appended claims and their equivalents. In other words, the above detailed description is based on the specific examples and possible implementations of the present disclosure, but it should not be interpreted so that only explicitly disclosed implementations are feasible. It is intended that any change or substitution that may be made to the disclosure by an ordinary skill person without introducing a creative and / or technical contribution to the technology should fall within the scope of protection taking into account equivalents, which is provided by the following claims of the present disclosure.
Claims
1.A method for synthesizing a video, the method including:obtaining a shared motion latent code corresponding to a motion between frames of the video by inputting an input frame to a first encoder;obtaining a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame; andpredicting, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation.2.The method of claim 1, wherein the first encoder includes a Contrastive Language-Image Pretraining (CLIP) encoder; wherein the second encoder includes pre-trained Variational Autoencoder (VAE); wherein the trained neural network model includes a U-Net-based diffusion predictor.3.The method of claims 1 or 2, wherein the predicting the next frame includes:predicting, by using the trained neural network model, one more next frame from the previously predicted next frame based on the shared motion latent code and the first latent representation.4.The method of any one of claims 1 to 3, wherein the motion between the frames of the video being synthesized is based on the shared motion latent code by inputting the shared motion latent code to one or more layers of the trained neural network model.5.The method of any one of claims 1 to 4, wherein the obtaining the shared motion latent code includes:obtaining a Contrastive Language-Image Pretraining (CLIP) embedding for the input frame by inputting the input frame to the first encoder;identifying one or more CLIP embeddings nearest to the obtained CLIP embedding using a k-Nearest Neighbors (kNN) and a distance metric for the kNN, based on a database of CLIP embeddings obtained for one or more frames of one or more training videos comprised in a training dataset used for training the neural network model; andobtaining the shared motion latent code from a bank of learned motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more CLIP embeddings.6.The method of claim 5, wherein the distance metric for the kNN is computed as the distance of the CLIP embedding of a frame of a training video comprised in the training dataset and the CLIP embedding of the input frame.7.The method of any one of claims 1 to 6, wherein the trained neural network model is trained by repeatedly performing:obtaining a pair of frames including the input frame and the next frame from a training video of a training dataset;obtaining the first latent representation and a second latent representation of the pair of frames by inputting the pair of frames to the second encoder;computing a residual by subtracting the first latent representation from the second latent representation;obtaining the shared motion latent code of the training video indicated by the index, from which the pair of frames are obtained, and storing the obtained shared motion latent code to the database of CLIP embeddings;obtaining a diffusion timestep index corresponding to the ordinal of the diffusion timesteps, wherein the diffusion timestep index is drawn into the neural network model based on sine or cosine function;obtaining a sampled noise tensor by sampling a noise tensor based on a normal distribution;obtaining a predicted noise tensor based on the shared motion latent code, the diffusion timestep index, the first latent representation, and the residual; andupdating, based on the sampled noise tensor, the predicted noise tensor, weights of the neural network model.8.The method of claim 7, wherein the obtaining a predicted noise tensor includes:obtaining the predicted noise tensor by inputting the shared motion latent code, the diffusion timestep index, the first latent representation, and the computed residualto the neural network model.9.The method of any one of claims 7 to 8, wherein the pair of frames obtained from the training video are frames located adjacently or near to each other in the sequence of frames of the training video,wherein the updating the weights of the neural network model includes minimizing a loss function between the predicted noise tensor and the sampled noise tensor by using a gradient descent.10.The method of any one of claims 7 to 9, wherein the obtaining the first latent representation and the second latent representation includes:obtaining crops of each of the pair of frames including a center and one or more of four corners of the pair of frames; andobtaining the first latent representations and the second latent representation of the pair of frames by inputting the pair of frames to the second encoder in the form of the crops of the pair of frames.11.The method of any one of claims 7 to 10, wherein the obtaining the pair of frames further includes:obtaining additional pairs of frames by performing at least one of horizontal flip or time inversion of the pair of frames from the training videos comprised in the training dataset.12.The method of any one of claims 1 to 11, wherein predicting the next frame from the input frame includes: obtaining the CLIP embedding for the input frame;identifying one or more CLIP embeddings nearest to the obtained CLIP embedding using kNN, based on the database of the CLIP embeddings obtained previously for one or more training videos comprised in the training dataset used for training the neural network model;obtaining the shared motion latent code based on a bank of learned motion embeddings corresponding to one or more frames of the one or more training videos that were used to obtain the one or more CLIP embeddings;obtaining the first latent representation by inputting the input frame to the second encoder;for each frame in the video being synthesized by using the trained neural network model starting with the next frame:obtaining an initial noised residual;obtaining a denoised residual by performing steps of a diffusion by the trained neural network model, wherein for each the steps of the diffusion include: subtracting from a noised residual an output value of the trained model;obtaining the second latent representation of a next frame by adding the denoised residual to the first latent representation of the input frame;obtaining the next frame by inputting the second latent representation to a decoder corresponding to the second encoder; andobtaining the synthesized video by concatenating all obtained frames.13.An electronic device for synthesizing a video the electronic device comprising:a memory storing computer-executable program andat least one processor;wherein the at least one processor execute the program or at least one instruction stored in the memory to cause the electronic device to:obtain a shared motion latent code associated with a motion between frames of the video by inputting an input frame to a first encoder;obtain a first latent representation by inputting the input frame to a second encoder to reduce a resolution of the input frame; andpredict, by using a trained neural network model, a next frame from the input frame based on the shared motion latent code and the first latent representation.14.The electronic device of claim 13, wherein the first encoder includes a Contrastive Language-Image Pretraining (CLIP) encoder; wherein the second encoder includes pre-trained Variational Autoencoder (VAE); wherein the trained neural network model includes a U-Net-based diffusion predictor.15.A computer-readable medium storing computer-executable instructions which when executed by an electronic device cause the electronic device to perform the method of any one of claims 1 to 12.
Citation Information
Patent Citations
A method for detecting abnormal behavior in videos
CN108647649B
Learned B-frame compression
US11399198B1
Generating video frames using neural networks
US20210019555A1
Method for video compression processing, electronic device and storage medium
US20210044804A1
Method and apparatus for substitutional neural residual compression
US20220020182A1
Cited By
Universal image restoration method and system based on double-space joint diffusion
CN120495139A
A universal image restoration method and system based on dual-space joint diffusion
CN120495139B
Wireless VR streaming super-division and frame prediction optimization method based on visual model
CN121967723A
Video frame generation method and apparatus, and device training method and apparatus
CN122824921A
