Customizing actions and appearances in video generation

Abstract

Embodiments of the present disclosure relate to customizing actions and appearances in video generation. Systems and methods include generating a synthesized video based on a custom action. A video generation system obtains a textual cue that includes an object and a custom action lexical element. The custom action lemma represents a custom action. The system encodes the text cues to obtain text embedding. The video generation model then uses the video generation model to generate a synthesized video depicting the object performing the custom action based on text embedding.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 594,328, filed on October 30, 2023, and U.S. Patent Application No. 18 / 584,210, filed on February 22, 2024. The entire contents of the above applications are incorporated herein by reference for all purposes. Background Art

[0003] The present disclosure relates generally to image processing, and more specifically to video generation. Image processing is a type of data processing that involves the manipulation of images to obtain a desired output, and image processing generally utilizes specialized algorithms and techniques. Image processing is used to perform operations on images to enhance the quality of the images or to extract useful information from the images. The process generally includes a series of steps, including the input of an image, analysis of the image, manipulation to enhance features or remove noise, and a final output of the enhanced image or salient information contained in the image.

[0004] Image processing techniques are also used for image generation. For example, machine learning (ML) techniques have been applied to create generative models that can generate new image content. One use for generative artificial intelligence is to create images based on input prompts. This task is often referred to as a "text to image" task or simply "text2img", although images, videos, and other types of prompts can be used. ML models including generative adversarial networks (GANs) and variational autoencoders (VAEs) have been adapted to generate pixel data to create novel images. Newer methods (such as denoising diffusion probability models (DDPMs)) iteratively refine the generated images based on guidance (such as text prompts). Recently, image generation models have been adapted for use in video generation by using models to generate video frames. Summary of the invention

[0005] Systems and methods for generating videos with customized actions are described herein. Embodiments include a video generation device configured to identify a customized action from a text prompt, and then generate a video including the customized action. In some cases, the customized action is identified by a nonce character (such as, "[V]") in the text prompt. A text encoder processes the text prompt to generate a text embedding including a customized action word. The customized action word is a vector representation of the concept of a customized action learned by a video generation model. The word is decoded by the video generation model during a generative process to generate a video. In some embodiments, the video generation model includes a diffusion model architecture. Some embodiments are further configured to generate a video including a customized appearance using a similar method.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for video generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include obtaining a text prompt including an object and a custom action token, wherein the custom action token represents a custom action; encoding the text prompt to obtain a text embedding; and using a video generation model to generate a synthesized video based on the text embedding, the synthesized video depicting the object performing the custom action.

[0007] A method, apparatus, non-transitory computer-readable medium, and system for video generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include obtaining a training set including a plurality of video clips, the video clips respectively depicting a plurality of different entities performing a custom action; and training a video generation model using the training set to generate a synthetic video depicting an arbitrary entity performing a custom action based on a custom action token in a text prompt. In some cases, obtaining the training set may include creating a custom training set or using a pre-existing training data set for training a machine learning model.

[0008] An apparatus, system, and method for video generation are described. One or more aspects of the apparatus, system, and method include at least one processor; at least one memory storing instructions executable by the at least one processor; and a video generation model including parameters stored in the at least one memory, and the video generation model is trained to generate a synthetic video depicting an arbitrary entity performing a custom action, wherein the custom action is identified by a custom action token within a text prompt. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of a video generation system according to aspects of the present disclosure is shown.

[0010] Figure 2 An example of a video generating apparatus according to aspects of the present disclosure is shown.

[0011] Figure 3 An example of a video generation pipeline according to aspects of the present disclosure is shown.

[0012] Figure 4 An example of an output video depicting a custom action is shown according to aspects of the present disclosure.

[0013] Figure 5 An example of an output video with customized motion and customized appearance according to aspects of the present disclosure is shown.

[0014] Figure 6 An example of a video generation model according to aspects of the present disclosure is shown.

[0015] Figure 7 An example of a method for providing customized videos to users according to aspects of the present disclosure is shown.

[0016] Figure 8 An example of a method for generating a video with customized motion according to aspects of the present disclosure is shown.

[0017] Fig. 9 An example of a method for training a diffusion-based video generation model according to aspects of the present disclosure is shown.

[0018] Fig.10 An example of a method for training a machine learning model according to aspects of the present disclosure is shown.

[0019] Fig.11 An example of a video generation device training pipeline according to aspects of the present disclosure is shown.

[0020] Fig.12 An example of a computing device according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Image processing is a fundamental aspect of computer vision, focusing on the manipulation, analysis, and interpretation of visual data. With the development of advanced algorithms and computing techniques, image processing has been extended to include image generation. In particular, generative models have enabled the synthesis of realistic images based on a variety of inputs.

[0022] Building on the foundations of image generation, the principles have been extended to video generation. In this context, adaptations such as temporal coding have been introduced to maintain coherence and continuity over sequential frames. While generative models have proven effective for image synthesis, the generation of consistent and realistic video sequences presents additional challenges due to the added temporal dimension. In addition, creators have worked hard to generate specific concepts in the synthesized video, such as the specific identity of an actor or object, or a specific action in the synthesized video.

[0023] There are some conventional systems and methods for synthesizing specific concepts in generated concepts. There are customized methods for text to image that focus on transferring appearance to generated images. However, the system is mainly designed for images, and therefore, does not meet the needs of video generation when motion and time consistency are necessary. Some controllable video generation systems have been tuned to learn specific human dances and transfer them to various actors, however, the system cannot learn universal motion concepts, and such learned motions cannot be reproduced by characters or non-human objects. Other methods include using a pre-trained text to image model to reverse existing videos frame by frame, and then synthesizing videos using the reversed noise map and text guidance. However, this is limited to editing existing videos and does not allow the creation of new videos.

[0024] Some methods learn a separate action model to generate an action prior, which involves a separate model and parameters, giving a considerable computational overhead. In addition, the action concepts learned by the action model are relatively simple and can generally only be transferred to another single object. Finally, some text-to-video models have been developed to add new topics to the base generative model. Although they demonstrate capabilities in learning appearance, the system cannot effectively learn or transfer action concepts, limiting their application in dynamic video generation.

[0025] Embodiments of the present disclosure improve upon video generation systems by enabling the generation of synthesized videos with custom motion concepts that can be applied to any actor or object. Traditional video generation models do not transfer custom motions to new objects, so using traditional tools to perform this task would involve expensive and time-consuming frame-by-frame editing, such as rotoscoping or otherwise using existing base video. In contrast, embodiments of the present disclosure quickly and efficiently generate videos with custom motions transferred to new objects.

[0026] Embodiments of the present disclosure include a video generation model that is tuned to learn both customized action concepts and customized appearance concepts without using a separate adapter model or adapter layer. The learned concepts are not spatially limited. For example, a dance can be learned during training, and the generated video can include multiple actors performing the same dance in different areas. The action concept is further not limited to actions performed by actors: in another example, camera movements that follow an object in a particular way can be learned and reproduced.

[0027] Reference Figures 1 to 6 A video generation system is described. Figures 7 and 8 A method for generating a video with customized motion is described. Figures 9 to 11 Methods for training machine learning models, including video generation models, are described. Fig.12 A computing device configured to implement a video generating apparatus is described.

[0028] Video Generation System

[0029] An apparatus for video generation is described. One or more aspects of the apparatus include at least one processor; at least one memory storing instructions executable by the at least one processor; and a video generation model including parameters stored in the at least one memory, and the video generation model is trained to generate a synthesized video and a custom action token representing a custom action based on a text prompt, wherein the synthesized video depicts the custom action.

[0030] Some examples of the apparatus, system and method also include a user interface configured to obtain text prompts. Some examples also include a text encoder configured to encode the text prompts to obtain text embeddings. In some embodiments, the video generation model includes a diffusion model. In some embodiments, the video generation model includes a transformer model.

[0031] Some examples of the apparatus, system and method also include an embedding optimizer configured to contextualize the custom action token. Some examples of the apparatus, system and method also include a training component configured to train the video generation model based on the training data.

[0032] Figure 1 An example of a video generation system according to aspects of the present disclosure is shown. The example shown includes a video generation device 100 , a database 105 , a network 110 , and a user interface 115 .

[0033] In the example process, the user provides a text input identifying a custom action. The custom action can be identified using a random number character (such as, "[V]" shown). Then, the video generation device 100 processes the text input to generate a synthesized video based on the text input. The synthesized video includes the custom action. The synthesized video can also include other details from the prompt. Therefore, the system enables users to generate videos depicting custom actions performed by various actors (including, humans, animals, characters, etc.).

[0034] An embodiment of the video generating device 100 includes a component implemented on a server. The server provides one or more functions to a user linked by means of one or more networks in various networks. In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses a microprocessor and a protocol via a hypertext transfer protocol (HTTP) and a simple mail transfer protocol (SMTP) to exchange data with other devices / users on one or more networks in the network, although other protocols (such as file transfer protocol (FTP) and simple network management protocol (SNMP)) may also be used. In some cases, the server is configured to send and receive hypertext markup language (HTML) formatted files (e.g., for displaying web pages). In various embodiments, the server includes a general computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0035] The video generating apparatus 100 may also have one or more components implemented in a user device. In other words, the video generating apparatus 100 may include modules including instructions stored and / or executed on a user device, and other modules including instructions stored and / or executed on a server. In some cases, the modules are partitioned into another server, such as a server dedicated to training a model included in the video generating apparatus 100, or a server dedicated to adjusting text embeddings. The video generating apparatus 100 is a reference to Figure 2 Examples of corresponding elements described, or including reference Figure 2 Aspects of the corresponding elements described.

[0036] Database 105 stores information used by the video generation system. For example, the database can store previous prompts, previously generated videos, video generation model parameters, training data, user configuration settings, etc. Database 105 is an organized batch of data. For example, database 105 stores data in a specified format known as a schema. The database can be constructed as a single database, a distributed database, a plurality of distributed databases, or an emergency backup database. In some cases, a database controller can manage data storage in the database and processing in the database. In some cases, a user interacts with the database controller. In other cases, the database controller can automatically operate without user interaction.

[0037] The network 110 facilitates (e.g., via a user interface 115) information transfer between the video generation device 100, the database 105, and the user, etc. In some cases, the network is referred to as a "cloud". A cloud is a computer network configured to provide on-demand availability of computer system resources (such as data storage and computing power, etc.). In some examples, the cloud provides resources without being actively managed by the user. The term cloud is sometimes used to describe a data center that is available to many users on the Internet. Some large cloud networks have functions distributed in multiple locations from a central server. If a server has a direct or close connection to a user, the server is designated as an edge server. In some cases, the cloud is limited to a single organization. In other examples, the cloud can be used for many organizations. In one example, the cloud includes a multi-layer communication network including multiple edge routers and core routers. In another example, the cloud is based on a batch of switches locally in a single physical location.

[0038] The user interface 115 enables a user to interact with the device. In some embodiments, the user interface 115 may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces with the user interface directly or through an IO controller module). In some cases, the user interface 115 may be a graphical user interface (GUI). For example, a GUI may be part of a web application, or part of a program (such as design document editing software).

[0039] According to some aspects, the user interface 115 obtains a text prompt indicating a custom action. For example, a user can enter a text prompt in a text field of the user interface. In some aspects, the text prompt includes a random number of characters representing the custom action. The user interface 115 is referenced Figure 2 Examples of the described corresponding elements, or aspects including the corresponding elements.

[0040] Figure 2 An example of a video generation apparatus 200 is shown according to aspects of the present disclosure. The example shown includes the video generation apparatus 200, a user interface 205, a text encoder 210, a video generation model 215, an embedding optimizer 220, and a training component 225.

[0041] The video generation device 200 is a reference Figure 1 Examples of corresponding elements described herein, or aspects including the corresponding elements. User interface 205 is a reference to Figure 1 The user interface 205 may be implemented as part of the video generating apparatus 200, or may be implemented in another device (such as, Figure 1The text encoder 210 is a reference to the user device shown in FIG. Figure 3 and Figure 6 The video generation model 215 is a reference to an example of a corresponding element described, or an aspect including the corresponding element. Figure 3 , Figure 4 and Fig.11 Examples of corresponding elements described, or aspects including the corresponding elements. Training component 225 is a reference Fig.11 Examples of the described corresponding elements, or aspects including the corresponding elements.

[0042] Embodiments of the video generating apparatus 200 include component parts that can be further divided into subcomponents. The components are named and described differently to partition the functions enabled by the processor and to partition the computing devices used in the video generating apparatus 200 (such as, for example, reference Fig.12 In some examples, the partitions are physically implemented, such as through the use of separate circuits or processors for each component. In some examples, the partitions are logically implemented via the architecture of code that can be executed by the processor.

[0043] The video generation device 200 includes a component including one or more artificial neural network (ANN) subcomponents. For example, the text encoder 210, the video generation model 215, and the embedding optimizer 220 all include ANN subcomponents. ANN is a hardware or software component including multiple connection nodes (i.e., artificial neurons), which loosely correspond to neurons in the human brain. Each connection or each edge transmits a signal from one node to another node (e.g., a physical synapse in the brain). When a node receives a signal, it processes the signal and then transmits the processed signal to other connected nodes. In some cases, the signal between nodes includes real numbers, and the output of each node is calculated by a function of the sum of its inputs. In some examples, the node can use other mathematical algorithms (e.g., selecting the maximum value as output from the input) or any other suitable algorithm for activating a node to determine the output of the node. Each node and each edge is associated with one or more node weights that determine how the signal is processed and transmitted.

[0044] During the training process, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). The weight of the edge increases or decreases the strength of the signal transmitted between the nodes. In some cases, the nodes have a threshold below which the signal is not transmitted at all. In some examples, the nodes are clustered into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0045] The user interface 205 enables a user to interact with the video generation apparatus 200, and may include a display, an input / output device (such as a mouse and a keyboard), and one or more graphical user interface (GUI) elements displayed on the display. In some cases, the user interface 205 is implemented as part of a separate device (e.g., an edge device or a user device).

[0046] The text encoder 210 is configured to generate text embeddings from text prompts. Text embedding is a vector representation of text that captures meaning from input text. An embodiment of the text encoder 210 includes a transformer-based encoder (such as, Flan-T5 or CLIP text encoder). A transformer or transformer network is a type of neural network model for natural language processing tasks. The transformer network uses an encoder and a decoder to convert one sequence into another sequence. The encoder and decoder include modules that can be stacked multiple times on top of each other. The module includes a multi-head attention layer and a feedforward layer. The input and output (target sentence) are first embedded in an n-dimensional space. The position encoding of different words (i.e., giving each word / part in the sequence a relative position because the sequence depends on the order of its elements) is added to the embedding representation (n-dimensional vector) of each word. In some examples, the transformer network includes an attention mechanism, where the attention looks at the input sequence and decides which other parts of the sequence are important at each step.

[0047] The attention mechanism involves queries, keywords, and values, represented by Q, K, and V, respectively. Q is a matrix containing the query (a vector representation of a word in a sequence), K is all the keys (a vector representation of all the words in a sequence), and V is the value, which is also a vector representation of all the words in the sequence. For encoder and decoder, multi-head attention modules, V consists of the same sequence of words as Q. However, for attention modules that consider encoder and decoder sequences, V is different from the sequence represented by Q. In some cases, the values ​​in V are multiplied by some attention weights a and summed.

[0048] Embodiments of the text encoder 210 are not limited to the Flan-T5 encoder or the CLIP text encoder, and may include any encoder configured to generate a vector representation of text. The text encoder 210 identifies a custom action word element corresponding to a custom action. The custom action word element may be a text embedding described above or may be a part of a text embedding described above.

[0049] The video generation model 215 is configured to generate a synthesized video. Embodiments of the video generation model 215 include an image generation model (such as a diffusion model) that is adapted to generate temporally coherent frames in a sequence. For example, the video generation model 215 may include a diffusion model with an additional time layer. According to some aspects, the video generation model 215 uses the video generation model 215 to generate a synthesized video based on a text prompt and a custom action word, wherein the synthesized video depicts a custom action. In some embodiments, the video generation model 220 includes a generative transformer model.

[0050] In some aspects, the video generation model 215 is tuned to depict custom actions based on custom action tokens. For example, the embodiment video generation model 215 may include a pre-trained video generation model trained on a large data set, and the pre-trained video generation model uses the training data depicting the custom action to be tuned to learn the custom action as a new concept. According to some aspects, although the "custom action token" corresponds to the part of the text embedding produced by the text encoder 210, the embodiment is not limited thereto. In at least one embodiment, the custom action token corresponds to one or more words in the input text prompt, and the word is extracted as a custom token by a tokenizer.

[0051] Some embodiments of the video generation device 200 adjust the text embedding from the text encoder 210 so that the generated synthetic video has higher quality. For example, the embedding optimizer 220 can be trained to adjust the text embedding at the time of inference by changing the value within the text embedding. As used herein, training can be synonymous with "fine-tuning" or "tuning". During this training, the embedding optimizer 220 can learn general conversions applied to any and all text embeddings. In some cases, each embedding has a custom conversion applied to the embedding at the time of inference based on a specific text embedding. In some cases, the general conversion is applied to all text embeddings, and the fine-tuning conversion unique to each text embedding is further selectively applied. According to some aspects, the embedding optimizer 220 adjusts the text embedding while the parameters of the video generation model 215 are fixed.

[0052] According to some aspects, the embedding optimizer 220 optimizes the custom action token. In some examples, the embedding optimizer 220 contextualizes the custom action token based on the text prompt, wherein the synthesized video is based on the contextualized custom action token. For example, the contextualization can include adjusting the embedding of each word in the text prompt based on other embeddings of the remaining words to increase semantic alignment. The adjustment can be performed by a training model (such as a transformer-based model) in the embedding optimizer 220.

[0053] According to some aspects, the embedding optimizer 220 optimizes the custom action tokens using the video generation model 215. For example, the embedding optimizer 220 can evaluate the output from the video generation model 215 and then optimize the custom action tokens based on the difference from the expected output. In some examples, the embedding optimizer 220 contextualizes the custom action tokens based on the textual prompts, wherein the synthesized video is based on the contextualized custom action tokens. In at least one embodiment, the embedding optimizer 220 is implemented on a device different from the video generation device 200.

[0054] In some examples, the training component 225 augments the training data by generating one or more additional video segments based on the video segment set. In some examples, the training component 225 augments the training data by generating a set of captions for each video segment in the video segment set. In some aspects, the training component 225 uses a reconstruction loss and a regularization loss to tune the video generation model 220, wherein the regularization loss reduces the association between the custom action and one or more objects that perform the custom action in the video segment set. In some aspects, the training component 225 tunes the video generation model 220 by fixing one or more appearance layers and updating one or more time layers. In some cases, the appearance layer generates a potential embedding that is more influential on the spatial features and structural features of the generated video, while the time layer generates a potential embedding that is more influential on the action features of the generated video. In at least one embodiment, the training component 225 is implemented on a device different from the video generation device 200.

[0055] Figure 3 An example of a video generation pipeline according to aspects of the present disclosure is shown. The example shown includes a text prompt 300, a text encoder 310, a noise sample 315, a video generation model 320, and an output video 325.

[0056] Text prompt 300 is a reference Figure 6 Examples of corresponding elements described herein, or aspects including the corresponding elements. The text encoder 310 is a reference Figure 2 and Figure 6 The video generation model 320 is a reference to an example of a corresponding element described in the embodiment of the present invention, or an aspect including the corresponding element. Figure 2 , Figure 4 ,and Fig.11 Examples of the described corresponding elements, or aspects including the corresponding elements.

[0057] In this example, the system receives a text prompt 300 indicating a custom action. The user can provide the text prompt 300 (e.g., via a user interface), or the text prompt 300 can be obtained in another way (such as from another automated system). In this example, the text prompt 300 uses a random number character 305 "[V]" to indicate a custom action, i.e., "[V] gesture". The text encoder 305 then processes the text prompt 300 to generate a text embedding. The text embedding may include a custom word element corresponding to the custom action. For example, the text embedding may include a vector in the embedding space corresponding to "[V]" in the text, where the vector is a custom word element.

[0058] A noise component or noise sampler can generate noise samples 315, which are a series of pure noise images. The sampler can initially sample from, for example, a pure Gaussian noise distribution. Then, a video generation model 320 generates an output video 325 based on the noise samples 315 and text embeddings, the text embeddings including custom action tokens. According to some aspects, the pre-training of the video generation model 320 enables the video generation model 320 to generate custom actions in the output video 325 when conditioned by the custom action tokens in the text embedding space.

[0059] In some cases, at inference time, the video generation model samples the noise in a specific way. For example, a diffuse denoising process is defined over diffuse time steps t∈{0,1,2,...,T}, where x T is pure Gaussian noise and x0 is an image. When applying Eq. (2), some embodiments use the skewed distribution The sampling time step is replaced by a uniform sampling time step. In some cases, the earlier stages of the diffusion process have more influence on the structure of the video, while the later stages contribute mainly to fine-grained details because the image is almost fully formed. In some cases, the parameter α increases the skewness of the function and is set to α=0.5. For additional details on the reverse diffusion process (e.g., denoising), see Figure 6 is provided.

[0060] Figure 4 An example of an output video 410 depicting a custom action according to aspects of the present disclosure is shown. The example shown includes a training clip 400, a video generation model 405, and an output video 410. The training clip 400 is a reference to Fig.11 The video generation model 405 is a reference to an example of a corresponding element described, or an aspect including the corresponding element. Figure 2 , Figure 3 ,and Fig.11 Examples of the described corresponding elements, or aspects including the corresponding elements.

[0061] In this example, the training data includes a real (i.e., not synthesized or generated) segment training segment 400 that includes an actor performing an [X] action (e.g., an "air quote" action). The user can provide a text prompt indicating a custom action. The text prompt used in this example can be "person performing [X] action". The video generation model 405 then generates an output video 410 depicting a person not in the training data, thereby generalizing the action to the new actor.

[0062] Figure 5 An example of an output video with customized motion and customized appearance according to aspects of the present disclosure is shown. The example shown includes a training clip 500 with customized appearance, a training clip 505 with customized motion, and an output 510 with customized motion and customized appearance.

[0063] In some cases, the video generation system can learn a customized appearance concept and a customized action concept. The video generation system can be trained based on one or more loss functions using training clips 500 with customized appearance and training clips 505 with customized action. Fig.11 Additional details about training including loss functions are described. In this example, the custom appearance is linked to a random number character "[V]", and the custom action is linked to a random number character "[X]". The user can then invoke both the custom appearance concept and the custom action concept by inserting the two random number characters into a text prompt (e.g., "A [V] man doing the [X] on a beach"). The video generation system will then generate an output 510 with a custom action and a custom appearance including both the appearance concept and the action concept.

[0064] Figure 6 An example of a video generation model according to aspects of the present disclosure is shown. The example shown includes a guided latent diffusion model 600, an original image 605, a pixel space 610, an image encoder 615, original image features 620, a latent space 625, a forward diffusion process 630, noise features 635, a backward diffusion process 640, denoised image features 645, an image decoder 650, an output image 655, a text prompt 660, a text encoder 665, a guided feature 670, and a guided space 675. The text prompt 660 is a reference to Figure 3 Examples of corresponding elements described, or aspects including the corresponding elements. The text encoder 665 is a reference Figure 2 and Figure 3Examples of the described corresponding elements, or aspects including the corresponding elements.

[0065] In the example process, the guided latent diffusion model 600 processes the original image 605 in the pixel space 610 using the image encoder 615 to generate the original image features 620 in the latent space 625. The pixel space includes height, width, and color channel dimensions, while the latent space may include different or higher dimensions than the pixel space. The embedding in the latent space captures additional relationships, such as semantic information, structural features, and local and global influences. The forward diffusion process 630 iteratively adds noise to the original image features 620 to produce noise features 635. The reverse diffusion process 640 iteratively removes noise to produce denoised image features 645. The denoised image features 645 are decoded using the image decoder 650 to produce an output image 655 in the pixel space 610. In some cases, the reverse diffusion process 640 is guided using the guided features 670 in the guided space 675, which may be the same or different from the latent space 625. For example, the text encoder 665 may encode the text prompt 660 to produce the guided features 670. The guided features 670 are also vector representations that can be combined with or referenced by the embeddings at various stages of the back diffusion process 640 to regulate the generation of images and videos.

[0066] The example shown is a video generation model based on latent diffusion. A general guided latent diffusion model will be described below. Such models are typically used for image generation, but because videos include sequences of images, such models can also be intuitively used for video generation. In some examples, the diffusion model is based on a neural network structure called U-Net. U-Net takes input features with an initial resolution and an initial number of channels, and processes the input features using an initial neural network layer (e.g., a convolutional network layer) to produce intermediate features. The intermediate features are then downsampled using a downsampling layer so that the downsampled features have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

[0067] This process is repeated multiple times and then the process is reversed. That is, the downsampled features are upsampled using the upsampling process to obtain upsampled features. The upsampled features can be combined with intermediate features having the same resolution and the same number of channels via skip connections. The input is processed using the final neural network layer to produce output features. In some cases, the output features have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0068] In some cases, the U-Net takes additional input features to produce conditionally generated outputs. For example, the additional input features may include a vector representation of the input prompt. The additional input features may be combined with intermediate features within the neural network at one or more layers. For example, a criss-cross attention module may be used to combine the additional input features with the intermediate features.

[0069] The diffusion process can also be modified based on conditional guidance. In some cases, the user provides a text prompt that describes the content to be included in the generated image. For example, the user can provide a prompt such as, "Mouse that performs [V] action." In some examples, the guidance can be provided in a form other than text, such as via an image, sketch, or layout. The system converts the text prompt (or other guidance) into a conditional guidance vector or other multidimensional representation. For example, a transformer model or a multimodal encoder can be used to convert text into a vector or a series of vectors. In some cases, the encoder for conditional guidance is trained independently of the diffusion model.

[0070] A noise map comprising random noise is initialized. The noise map can be in pixel space or latent space. By initializing an image with random noise, different variations of an image can be generated that include content described by a conditional guidance. The system then generates an image based on the noise map and the conditional guidance vector.

[0071] The diffusion process may include both a forward diffusion process for adding noise to an image (or a feature in a latent space) and a backward diffusion process for denoising the image (or feature) to obtain a denoised image. The forward diffusion process may be represented as q(x t ∣x t-1 ), and the reverse diffusion process can be expressed as p(x t-1 ∣x t ). In some cases, a forward diffusion process is used during training to generate images with successively larger noise, and the neural network is trained to perform a backward diffusion process (ie, successively removing noise).

[0072] In the example forward process for the latent diffusion model, the model uses a Markov chain to map the observed variable x0 (in pixel space or latent space) to intermediate variables x1,…,x T When the latent variable passes through a neural network (such as U-Net), the Markov chain gradually adds Gaussian noise to the data to obtain an approximate posterior q(x 1：T |x0), where x1,…,x T has the same dimensions as x0. The forward diffusion process is generally associated with the training of latent diffusion models. Fig. 9An example method for training a latent diffusion model is described.

[0073] Neural networks can be trained to perform the reverse process. The back diffusion process is the basis for image and video generation. During the back diffusion process, the model is fed with noisy data x T (such as a noisy image), and denoise the data to obtain p(x t-1 ∣x t ). At each step t-1, the reverse diffusion process takes x t (such as the first intermediate image) and t as input. Here, t represents the step in the transformation sequence associated with different noise levels. The back diffusion process iteratively outputs x t-1 (such as the second intermediate image) until x T is restored back to the original image x0. The reverse process can be expressed as:

[0074] p θ (x t-1 |x t ):=N(x t-1 ;μ θ (x t ,t),∑ θ (x t ,t)). (1)

[0075] The joint probability of a sample sequence in a Markov chain can be written as the product of the conditional probability and the marginal probability:

[0076]

[0077] Because the reverse process takes the result of the forward process, that is, the sample of pure noise, as input, p(x T )=N(x T ; 0, I) is a pure noise distribution, and Represents a sequence corresponding to the Gaussian transform of a Gaussian noise sequence appended to the samples.

[0078] At the interference time, the observation data x0 in the pixel space can be mapped into the latent space as input, and the generated data Mapping from the latent space back to the pixel space as the output. In some instances, x0 represents the original input image with low image quality, and the latent variables x1,…,x T represents a noisy image, and Represents a generated image with high image quality.

[0079] Generate videos with custom motion

[0080] A method for video generation is described. One or more aspects of the method include obtaining a text prompt indicating a custom action; identifying a custom action token corresponding to the custom action; and using a video generation model to generate a synthesized video based on the text prompt and the custom action token, wherein the synthesized video depicts the custom action.

[0081] Some examples of the method, apparatus, non-transitory computer readable medium, and system also include identifying a custom object word-gram representing a custom object, wherein a synthesized video is generated based on the custom object word-gram and depicts the custom object. In some aspects, the video generation model is trained to depict the custom action based on the custom action word-gram. In some aspects, the text prompt includes a random number character representing the custom action.

[0082] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include optimizing the custom action word using a video generation model or a dedicated embedding optimizer component. Some examples also include contextualizing the custom action word based on a text prompt, wherein the synthesized video is based on the contextualized custom action word.

[0083] Figure 7 An example of a method 700 for providing a customized video to a user according to aspects of the present disclosure is shown. In some examples, the operation is performed by a system including a processor that executes a code set to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, the operation is performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed together with other operations.

[0084] At operation 705, the user provides a text prompt indicating a custom action. The user can Figure 1 to Figure 2 In this example, the user enters "Fluffy Persian cat performing [V] dance", where "[V]" is a custom identifier in the form of random number characters and represents a custom dance in which the actors swing their arms back and forth and step left and right.

[0085] At operation 710, the system encodes the text input to generate a text embedding, optionally optimizing the text embedding. The system may use, for example, reference Figure 2 The described text encoder encodes a text input and then optimizes the text embeddings using an embedding optimizer. According to some aspects, optimizing the embeddings can include a contextualization operation that updates the embedding of each word in the text input based on the embeddings of other words.

[0086] At operation 715, the system generates a customized video including the customized action. According to some aspects, this operation is performed by a video generation model trained to reproduce the customized action associated with the customized identifier. Figure 4 Additional details about training are described.

[0087] Figure 8 An example of a method 800 for generating a video with customized actions according to aspects of the present disclosure is shown. In some examples, the operation is performed by a system including a processor, which executes a code set to control the functional elements of the device. Additionally or alternatively, some processes are performed using dedicated hardware. Generally, the operation is performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed together with other operations.

[0088] At operation 805, the system obtains a text prompt indicating a custom action. The text prompt may further include an object, for example, an object designated to perform the custom action. In some cases, the operation of this step refers to a reference to Figure 1 and Figure 2 The user interface described may be executed by the user interface. Although the embodiment is not limited thereto, the custom action may be indicated by a random number word element representing the custom action. For example, the custom action may be indicated by natural language.

[0089] At operation 810, the system encodes the text hint to obtain a text embedding. In some cases, the operation of this step refers to reference Figure 2 , Figure 3 ,and Figure 6 The text encoder described herein may be executed by the text encoder. The term "custom action token" may refer to a random number character from a text prompt, or may refer to an embedding of a random number character generated by a text encoder.

[0090] At operation 815, the system generates a synthesized video based on the text prompt and the custom action token using the video generation model, wherein the synthesized video depicts the custom action. For example, the synthesized video may include an object from the text prompt performing the custom action. In some cases, the operation of this step refers to reference Figures 2 to 4 ,and Fig.11The video generation model described herein may be, or may be performed by, the video generation model. According to some aspects, the video generation model is tuned during a training phase to generate a video based on the custom action tokens, the video depicting an arbitrary entity performing the custom action. For example, the entity may be a human model, a humanoid character, an animal character, an anthropomorphic object, or some other actor. For example, the entity may be a non-human entity performing a human action (such as a dance). In another example, the entity may be a non-human entity, such as a car performing a custom driving action. Figures 9 to 11 Describes the training of a video generation model.

[0091] train

[0092] Methods for video generation are described. One or more aspects of the method include obtaining training data including a plurality of video clips depicting custom actions, and using the training data to train a video generation model to generate a synthesized video based on a text prompt and a custom action token representing the custom action, wherein the synthesized video depicts the custom action. Some examples also include training the video generation model to generate a synthesized video based on a custom object token representing a custom object.

[0093] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include augmenting the training data by generating one or more additional video clips based on the plurality of video clips. Some examples further include augmenting the training data by correspondingly generating a plurality of captions for each of the plurality of video clips. Some examples further include tuning the video generation model using a reconstruction loss and a regularization loss, wherein the regularization loss reduces the association between the customized action and one or more objects performing the customized action in the plurality of video clips. Some examples further include fixing one or more appearance layers and updating one or more temporal layers.

[0094] Some embodiments of the present disclosure use both appearance and motion to customize the video generation model. is indicated, where Shows the subject or object that will be added to the model. Action by video The training data may include appearance examples D s and Action Example D m , and the corresponding text description C s , C m The actions can be performed by different objects across different backgrounds, and the commonality within the video is only the dynamic motion. s , C mIt can be a general sentence, such as "photo of person [V]" (for the appearance set) or "person doing [X]" (for the action). In some cases, a mapping is made between identifiers (e.g., [V]) and unused or less used words to guide the text embedding to a cleaner area in the embedding space. An example of a unique text token is "pll", such as "[V]", etc.

[0095] In order to compare with the example set (D M , C M ) defines the action to customize the pre-trained text-to-video model θ, the objective function is minimized, as expressed by the following formula:

[0096]

[0097] In some embodiments of the present disclosure, the objective function is used To train a video generative model with parameters θ, where (x, c) is a video-text pair. The video is represented by a latent vector derived from a latent diffusion model (LDM), which is a diffusion model that operates in latent space rather than pixel space. In some cases, rather than learning the denoising process in a complex high-dimensional space, the image is first encoded into a lower-dimensional vector through a learned encoder-decoder model. In some cases, the text c is conditioned t is encoded by the text embedding from a pre-trained text encoder. The initial noise map is Diffusion time step The noise latent sample can be obtained by The prediction from the noise is calculated, where x is the original image and w t is a user-defined variable that controls the sample quality and α t controls the amount of noise added at each diffusion time step and is defined according to a noise scheduler. The model is trained (on top of the pre-training of the base model) with the following objective function:

[0098]

[0099] In some embodiments, using a pair (x, c)∈{(D s , C s ), (D m , C m )} to fine-tune the parameters of the model. In some cases, the model is fine-tuned sequentially first with respect to the action, then with respect to the object, or vice versa, to customize the model with respect to both the action and the appearance.

[0100] In some cases, the training data is split into a training set and a regularization set. Regularization can help the training process by preventing the video generation model from forgetting concepts it has learned in the pre-training phase and by preventing the video generation model from strongly associating the concept of action with the concept of appearance. For example, regularization can prevent the video generation model from associating the concept of action with a specific actor and a specific background.

[0101] In some embodiments, a regularization set is applied to avoid information loss during optimization (e.g., when using a specific person to optimize the objective function, the model can "forget" the presence of other people in the input) and ensure that the knowledge is preserved. Some embodiments use real image and text pairs for the regularization set instead of using generated videos, which would reduce the quality of the customized model. The objective function is expressed as follows:

[0102]

[0103] Where (x i , c i ) from the appearance example set or the action example set D M , C M And (x j , c j ) from the regularized set D R , C R .

[0104] Fig. 9 An example of a method 900 for training a diffusion-based video generation model according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a code set to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, the operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein consist of various sub-steps or are performed together with other operations.

[0105] The diffusion model can be trained using a forward diffusion process and a backward diffusion process. In one example, a user initializes an untrained model at operation 905. Initialization can include defining the architecture of the model and establishing initial values ​​for model parameters. In some cases, initialization can include defining hyperparameters, such as the number of layers, the resolution of each layer block, and the channels of each layer block, the location of skip connections, and the like.

[0106] Then, in N stages in operation 910, the system then adds noise to the training image using a forward diffusion process. In some cases, the forward diffusion process is a fixed process in which Gaussian noise is continuously added to the image. In a latent diffusion model, Gaussian noise can be continuously added to features in the latent space.

[0107] At operation 915, each stage n starting from stage N uses a back diffusion process to predict the image or image features at stage n-1. For example, the back diffusion process can predict the noise added by the forward diffusion process, and the predicted noise can be removed from the image to obtain the predicted image. In some cases, the original image is predicted at each stage of the training process.

[0108] In operation 920, the training system compares the predicted image (or image features) at stage n-1 with the actual image (or image features) (such as the image at stage n-1 or the original input image). For example, given the observed data x, the diffusion model can be trained to compare the negative log-likelihood of the training data -log p θ (x) is minimized. Then, at operation 925, the training system updates the parameters of the model based on the comparison. For example, the parameters of the U-Net can be updated using gradient descent. The time-dependent parameters of the Gaussian transform can also be learned.

[0109] Fig.10 An example of a method 1000 for training a machine learning model according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a code set to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, the operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed together with other operations.

[0110] At operation 1005, the system creates a training set including video clips depicting a plurality of different entities performing custom actions. Examples of different entities include human models, cartoon characters, and other actors. In some embodiments, the entities consist of human models. In some cases, the operations of this step refer to reference Figure 2 and Fig.11 The training component described herein may be performed by or may be executed by the training component. The training component may be performed by or may be executed by Figure 2 The method may be implemented on the same video generating device as described, or may be implemented in a different system connected to the video generating device. In some cases, creating a training set may include obtaining a pre-existing training data set for training a machine learning model.

[0111] At operation 1010, the system uses the training set to train a video generation model to generate a synthetic video depicting an arbitrary entity performing a custom action based on the custom action tokens in the text prompt. In some cases, the operation of this step refers to reference Figure 2 and Fig.11 The system may update the video generation model by performing gradient descent based on the calculated loss described with reference to equations (3) to (5). According to some aspects, the training process enables the video generation model to generalize actions to different types of actors not presented in the training set. For example, even when the training set includes clips of human actors, the video generation model may generalize actions from the training set to generate a video of a non-human character performing the action.

[0112] Fig.11 An example of a video generation device training pipeline according to aspects of the present disclosure is shown. The example shown includes training clips 1100, training instructions 1105, training components 1110, augmented training data 1115, loss function 1120, video generation model 1125, and video output 1130.

[0113] Training segment 1100 is a reference Figure 4 Examples of corresponding elements described herein, or aspects including the corresponding elements. Training component 1110 is a reference to Figure 2 Examples of corresponding elements described herein, or aspects including the corresponding elements. The video generation model 1125 is a reference Figures 2 to 4 Examples of the described corresponding elements, or aspects including the corresponding elements.

[0114] In this example, training data including training segments 1100 and training captions 1105 are input to a training component 1110. The training component 1110 can use the training data directly, or can create augmented training data 1115 by applying various transformations to the training data. For example, the augmented training data 1115 can include visual distortions or color changes of the training segments 1100, as well as altered or additional training captions. In some embodiments, applying data augmentation improves the quality of the generated video. Data augmentation can be applied to image data / video data as well as example data sets {(D s , C s ), (D m , C m)}caption descriptions. For example, spatial enhancement and color enhancement are applied to image examples / video examples to improve generalization of objects. Some embodiments of the present disclosure augment the text description with descriptive adjectives about the person performing the action or details about the video (e.g., scene, location, etc.). The descriptive annotations are either formed manually, or by using an image caption system from frames in the video, or by using a large language model to paraphrase a given caption. The image caption system may include, for example, a language model (LM).

[0115] The training component compares the training data or the augmented training data with the video output 1130 generated by the video generation model 1125, and calculates the loss function 1120 based on the comparison. The loss function quantifies the difference from the video output 1130 and the training data, and can be calculated based on equations (3) to (5) above. Then, the training component 1110 updates the parameters of the video generation model 1125 based on the calculated loss (e.g., by back propagation). In some cases, some parameters of the video generation model 1125 are updated while other parameters remain fixed. For example, when learning an action concept, a subset of the training clips 1100 and the training subtitles 1105 corresponding to the action concept can be used, and the action layer of the video generation model 1125 can be updated while the appearance layer (e.g., spatial layer) remains fixed.

[0116] Fig.12 An example of a computing device 1200 is shown, according to aspects of the present disclosure. The example shown includes the computing device 1200, processor(s), memory subsystem 1210, communication interface 1215, I / O interface 1220, user interface component(s), and channel 1230.

[0117] In some embodiments, computing device 1200 is Figure 1 In some embodiments, the computing device 1200 includes one or more processors 1205 that can execute instructions stored in the memory subsystem 1210 to obtain a text prompt indicating a custom action; identify a custom action word-gram corresponding to the custom action; and use a video generation model to generate a synthesized video based on the text prompt and the custom action word-gram, wherein the synthesized video depicts the custom action.

[0118] According to some aspects, computing device 1200 includes one or more processors 1205. In some cases, the processor is an intelligent hardware device, for example, a general processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof. In some cases, the processor is configured to operate the memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor includes a dedicated component for modem processing, baseband processing, digital signal processing, or transmission processing.

[0119] According to some aspects, the memory subsystem 1210 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software including instructions, which, when executed, causes the processor to perform various functions described herein. In some cases, the memory includes, among others, a basic input / output system (BIOS), which controls basic hardware operations or software operations, such as, interactions with peripheral components or devices. In some cases, a memory controller operates a memory cell. For example, a memory controller may include a row decoder, a column decoder, or both decoders. In some cases, the memory cells in the memory store information in the form of logical states.

[0120] According to some aspects, the communication interface 1215 operates at the boundary between the communication entity (such as the computing device 1200, one or more user devices, the cloud, and one or more databases) and the channel 1230, and can record and process the communication. In some cases, the communication interface 1215 is provided to enable a processing system coupled to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna.

[0121] According to some aspects, I / O interface 1220 is controlled by an I / O controller to manage input and output signals of computing device 1200. In some cases, I / O interface 1220 manages peripheral devices that are not integrated into computing device 1200. In some cases, I / O interface 1220 represents a physical connection or port to an external peripheral device. In some cases, the I / O controller uses an operating system, such as, Or other known operating systems. In some cases, the I / O controller represents a modem, keyboard, mouse, touch screen, or similar device, or interacts with the device. In some cases, the I / O controller is implemented as a component of the processor. In some cases, the user interacts with the device via the I / O interface 1220 or via the hardware components controlled by the I / O controller.

[0122] According to some aspects, user interface component 1225 enables a user to interact with computing device 1200. In some cases, user interface component 1225 includes an audio device (such as an external speaker system), an external display device (such as a display screen), an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller), or a combination thereof. In some cases, user interface component 1225 includes a GUI.

[0123] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in the form of block diagrams to represent the relationships between components and avoid blurring the concepts described. Similar components or features may have the same name, but may have different reference numerals corresponding to different drawings.

[0124] Some modifications to the present disclosure may be readily apparent to those skilled in the art, and the principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should conform to the broadest scope consistent with the principles and novel features disclosed herein.

[0125] The described method can be implemented or executed by a device including a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof. The general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented by hardware or software, and can be executed by a processor, firmware, or any combination thereof. If the function is implemented in software executed by a processor, the function can be stored in a computer-readable medium in the form of instructions or codes.

[0126] Computer-readable media include non-transient computer storage media and communication media including any media that promote code or data transmission. Non-transient storage media can be any available media that can be accessed by a computer. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc (CD) or other optical disc storage device, magnetic disk storage device, or any other non-transient medium for carrying data or code or storing data or code.

[0127] In addition, the connection component can be appropriately referred to as a computer-readable medium. For example, if code or data are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals), then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology is included in the definition of the medium. The combination of media is also included in the scope of computer-readable media.

[0128] In the present disclosure and the appended claims, the word "or ( / or)" means an inclusive list, so that, for example, a list of X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. The phrase "based on" is also not used to indicate a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted to mean "based at least in part on". In addition, the words "a" or "an" indicate "at least one".

Claims

1. A method comprising: Obtaining a text prompt including an object and a custom action word, wherein the custom action word represents a custom action; encoding the textual cues to obtain text embeddings; as well as A synthetic video is generated based on the text embeddings using a video generation model, the synthetic video depicting the object performing the customized action.

2. The method according to claim 1, wherein: The text prompt includes a customized object word-gram representing a customized object; and The synthesized video depicts the customized object performing the customized action.

3. The method according to claim 1, wherein: The text prompt includes random number characters representing the customized action.

4. The method according to claim 1, further comprising: The custom action tokens are optimized using the video generation model.

5. The method according to claim 1, further comprising: The custom action word is modified based on the text prompt to obtain a contextualized custom action word, wherein the synthesized video is based on the contextualized custom action word.

6. The method according to claim 1, wherein: The video generation model is trained using a training set, the training set comprising a plurality of videos, the plurality of videos respectively depicting a plurality of entities performing the customized actions.

7. The method according to claim 6, wherein: The training set includes a plurality of videos, the plurality of videos respectively depicting a plurality of entities having a customized appearance, and wherein the customized appearance corresponds to a customized object token.

8. A method for training a machine learning model, the method comprising: obtaining a training set including a plurality of video clips, the plurality of video clips respectively depicting a plurality of different entities performing a customized action; as well as A video generation model is trained using the training set to generate a synthetic video depicting an arbitrary entity performing the custom action based on the custom action tokens in the text prompt.

9. The method of claim 8, wherein training the video generation model comprises: The video generation model is trained to generate a synthetic video depicting an entity having a customized appearance based on the customized object tokens in the text prompt.

10. The method according to claim 8, wherein obtaining the training set comprises: The training set is augmented by generating one or more additional video segments based on the plurality of video segments.

11. The method according to claim 8, wherein obtaining the training set comprises: The training set is expanded by generating one or more descriptions for each of the plurality of video clips, wherein each of the one or more descriptions includes the customized action token.

12. The method of claim 8, wherein training the video generation model comprises: The video generation model is tuned using a reconstruction loss and a regularization loss, wherein the regularization loss reduces the association between the customized action and one or more objects performing the customized action in the plurality of video clips.

13. The method of claim 8, wherein training the video generation model comprises: One or more appearance layers of the video generative model are fixed and one or more temporal layers of the video generative model are updated.

14. An apparatus comprising: at least one processor; at least one memory storing instructions executable by the at least one processor; as well as A video generation model includes parameters stored in the at least one memory and is trained to generate a synthesized video depicting an arbitrary entity performing a custom action, wherein the custom action is identified by a custom action token within a text prompt.

15. The apparatus according to claim 14, further comprising: A user interface is configured to obtain the text prompt.

16. The device according to claim 14, wherein: The video generation model includes a diffusion model.

17. The apparatus of claim 14, wherein: The video generation model includes a transformer model.

18. The apparatus according to claim 14, further comprising: A text encoder is configured to encode the text prompt to obtain a text embedding.

19. The apparatus according to claim 14, further comprising: An embedding optimizer is configured to contextualize the custom action word.

20. The apparatus of claim 14, further comprising: A training component is configured to train the video generation model based on training data.

Images