Controllable image-to-video generation

Abstract

Techniques for controllable image-to-video generation are generally described. In various examples, a first image representing at least a first object can be received. First input data including a selection of the first object in the first image for animation can be received. Second input data including at least a first bounding box indicating a target position of the first object can be received. A latent diffusion text-to-image model and the first image can be used to generate a first plurality of visual tokens. One or more first anchor tokens representing a position of the first bounding box can be generated. The latent diffusion text-to-image model can be used to generate a video animating the first object based on the first plurality of visual tokens and the one or more first anchor tokens.

Description

Background Technology

[0001] Artificial intelligence-based text-to-image and image-to-image models can be used to generate new images from input text and / or input images. In the context of text-to-image, the input is a natural language description, and the text-to-image model generates an image that matches the input description. Image-to-image models can take a source image as input and can generate a new image that incorporates characteristics of the target domain while preserving certain characteristics of the source image. In some cases, text descriptions can be used to impose conditions on the image-to-image model, describing the desired manipulations and / or transformations to be performed on the source image when generating a new image. Attached Figure Description

[0002] Figure 1 An example controllable image-to-video model is described, which allows a user to selectively and controllably animate one or more objects in an image according to various aspects of this disclosure.

[0003] Figure 2 Depicting various aspects of this disclosure Figure 1 The controllable image to the additional details of each component of the video model.

[0004] Figure 3 The invention describes the incorporation of a gated self-attention layer into a latent diffusion model, which can be used for misaligned motion generation, according to various aspects of this disclosure.

[0005] Figure 4 The various aspects described in this disclosure are as follows. Figure 1 The example architecture of the motion module described in [the document].

[0006] Figure 5 This is a block diagram illustrating an example architecture of a computing device that can be used according to various aspects of this disclosure.

[0007] Figure 6 This is a diagram illustrating an example system for sending and providing data that can be used according to this disclosure.

[0008] Figure 7 This is a block diagram illustrating an example process for controllable image-to-video generation according to embodiments of the present disclosure. Detailed Implementation

[0009] In the following description, reference is made to the accompanying drawings, which illustrate several examples of the invention. It should be understood that other examples may be utilized and various operational changes may be made without departing from the spirit and scope of this disclosure. Therefore, the following detailed description is not presented in a limiting sense, and the scope of embodiments of the invention is defined only by the claims of the granted patent.

[0010] The growing demand for controllable video generation underscores users' expectations of creating videos for an increasing list of applications, such as personalized content (e.g., for social media), educational content generation, dynamic visualizations through user-generated (and / or other) content, and entertainment content such as short films, which may require precise control over motion. Some recent approaches feed textual cues into generative AI models (e.g., diffusion-based text-to-image and / or text-to-video models), but such methods do not allow users to interactively control fine-grained details of animation (such as the trajectory or path of one or more objects in the video).

[0011] Some exemplary methods for controllable video generation can focus on the image-to-video (I2V) task. I2V starts with a given conditional image, thus eliminating the ambiguity often encountered in text-to-video (T2V) generation, enabling the generation of more diverse video animations based on additional conditions such as text, trajectories, or reference videos. Therefore, I2V positions itself as a promising direction for controllable video generation by combining accuracy, versatility, and a more user-friendly setup. Recent I2V methods utilize models trained on massive video datasets with pre-extracted motion features. However, challenges arise in terms of computational resources and data collection. For example, even with efficient training schemes such as efficient parameter fine-tuning, training the model to understand new control conditions (e.g., motion vectors, trajectories) remains a highly computationally intensive task. Regarding data collection, acquiring large amounts of video data with carefully labeled conditions for training can be time-consuming and costly.

[0012] This paper describes a novel machine learning architecture for I2V tasks, called... IVA 0 Providing zero-shot, controllable image-to-video animation without any I2V training data. Therefore, IVA... 0 This enables fine-grained user control over animation (e.g., via user-provided motion paths, sometimes called layouts or trajectories) without the need for any labeled training videos. In this context, zero-shot animation refers to IVA (Independent Visual Aid). 0 The ability of a model to perform I2V tasks without any training data (such as video data).

[0013] IVA 0 The input includes a conditional image (e.g., an image to be animated) and the motion trajectories of the object of interest, represented by a sequence of bounding box layouts (or trajectories drawn by the user on a graphical user interface). IVA 0The task is simplified by breaking down the controllable I2V task into two atomic tasks: (1) misaligned motion generation focuses on determining the rough layout of the object’s displacement throughout the video frame, and (2) in-place motion animation ensures the object’s consistency between frames while facilitating reasonable and smooth motion (pixel-level changes) across frames.

[0014] Figure 1 The controllable image-to-video model IVA is described. 0 As an example of 100, this model allows a user to selectively and controllably animate one or more objects in an image according to various aspects of this disclosure. For example... Figure 1 As shown, IVA 0 100 may include: (1) a pre-trained text-to-image model 122, (2) a control module (CM) 124, and (3) a motion module (MM) 126. wrong Position movement The generation is described as a layout-to-image generation task, which is achieved by inserting one or more gated self-attention layers as layout control modules and using bounding box layout for precise object placement. This is referred to as control module 124. transported in place move The animation is achieved by employing a temporal attention layer from the text-to-video generation task. This is referred to as Motion Module 126. Motion Module 126 maintains the consistency of the selected object by applying self-attention across frames, thereby achieving realistic and smooth transitions of the object from one frame to the next. Notably, CM 124 and MM 126 are pre-trained on the corresponding task-aligned dataset without any I2V-specific training. Additionally, this paper also describes a... Sports Disability Like inhibition The MAS (Motion-Assisted Frame) scheme generates frames by alternating between different fill operations to reduce the number of frames that might fall behind the motion trajectory. afterimage The illusion object, while maintaining the background.

[0015] The text-to-image model 122 can be a pre-trained latent diffusion imputation model (e.g., trained on a dataset that includes text and image pairs). Imputation is a digital image processing technique that can be used to replace a portion of an existing image with other image data. For example, imputation techniques can be used to restore or reconstruct missing or damaged parts of a photograph. In this case, imputation may involve replacing the relevant part of the image with new, plausible image data (e.g., pixel values) based on surrounding pixels or other information. In other examples, imputation may involve replacing a masked portion of an image with image data representing a target object (e.g., an object selected by the user). Ideally, imputation is performed so that the imputed object appears naturally within the image (e.g., based on primary lighting conditions, etc.).

[0016] Machine learning techniques, such as those described in this article, are commonly used to form predictions, solve problems, identify objects in image data for classification, and so on. For example, machine learning techniques can be used to detect objects represented in image data and / or translate text from one spoken language to another. In various examples, machine learning models can perform better than rule-based systems and can be more adaptive because the model can improve over time by retraining it as more data becomes available. Therefore, machine learning techniques often adapt to changing conditions. Deep learning algorithms, such as neural networks, are commonly used to detect patterns in data and / or perform tasks.

[0017] Generally, in machine learning models such as neural networks, parameters control the activation of neurons (or nodes) within a layer. The weighted sum of the activations of each neuron in the previous layer can be fed into an activation function (e.g., the sigmoid function, the rectified linear unit (ReLU) function, etc.). This result determines the activation of neurons in subsequent layers. Furthermore, bias values ​​can be used to shift the output of the activation function to the left or right on the x-axis, thus biasing the activation of neurons.

[0018] Generally, in machine learning models such as neural networks, after initialization, labeled training data is used to generate a cost or "loss" function that describes the difference between the expected and actual output of the machine learning model. The parameters of the machine learning model (e.g., weights and / or biases) can be updated to minimize (or maximize) the cost. For example, the machine learning model can use gradient descent (or ascent) algorithms to progressively adjust the weights so that the output of the loss function decreases (or increases) most rapidly. The method of updating the parameters of a machine learning model is often referred to as backpropagation.

[0019] A Transformer model (e.g., a Transformer machine learning model) is a machine learning model that includes an encoder network and a decoder network. The encoder takes the input and generates a feature representation of that input (e.g., feature vectors, feature maps, etc.). This feature representation is then fed into the decoder, which can generate the output based on the encoding. In natural language processing, Transformer models take sequences of words as input. For example, a Transformer can receive sentences and / or paragraphs that consist of sequences of words as input. In the various examples described in this paper, a Transformer can receive a set of object images as input. For example, a visual Transformer can be used to generate patches of the input image. Analogous to a natural language task, such image patches can act as “visual words.” Furthermore, with a visual Transformer, a backbone network is not required, and the raw pixel values ​​of the input image can be directly fed into the model.

[0020] Generally, the Transformer's encoder network consists of a set of encoding layers that process the input data layer by layer. Each encoder layer generates an encoding (referred to as a "token" in this paper). In the context of a visual Transformer, these tokens can be described as visual tokens. These tokens include feature representations (e.g., feature vectors and / or mappings) that include information about which parts of the input data are related to each other. For example, for each input embedding, the encoder layer can determine which parts of the token are related to other tokens received as part of the input data. Each encoder layer passes its token output to the next encoder layer. The Transformer's decoder network takes the tokens output by the encoder network and processes them using encoded contextual information and an encoder-decoder attention mechanism to generate an output embedding. Each encoder and decoder layer of the Transformer uses an attention mechanism that, for each input, measures the relevance of each other input and extracts information from the other inputs to generate the output. Each decoder layer also has an additional attention mechanism that extracts information from the output of the previous decoder before the decoder layer determines information from the encoding. The encoder and decoder layers have feedforward neural networks for additional processing of the output and include residual connections and layer normalization steps.

[0021] Scaling Dot Product Attention

[0022] The fundamental building block of the Transformer is the scaled dot product attention unit. As input data is passed to the Transformer model, attention weights between each word are computed simultaneously. The attention unit generates embeddings for each word in the context, which contain not only information about the word itself but also a weighted combination of other relevant words, weighted by the attention weights.

[0023] Specifically, for each attention unit, the Transformer model learns three weight matrices: query weights. W Q Key weights W K Sum weight W V For each lexical unit i Input embedding x i Multiply by each of the three weight matrices to produce the query vector. q i = x i W Q Key vector k i = x i W K Sum value vector v i = x i W V Attention weights are calculated using query vectors and key vectors: from lexical... i to word element j attention weights a ij It is q i and k j The dot product between them. The attention weights divided by the square root of the key vector dimension. This stabilizes the gradients during training. Then, the attention weights are passed to the softmax layer, which normalizes the weights so that their sum is 1. W Q and W K The fact that they are different matrices allows attention to be asymmetric: if the lexical units are different... i Focus on morphemes j This does not necessarily mean lexical units j Focus on lexical units i Attention units focus on lexical units. iThe output is a weighted sum of the value vectors of all words, with weights of . a ij That is, from i Attention is drawn to each word.

[0024] Attention calculation for all lexical units can be represented as a large matrix computation. Since matrix operations are optimized for fast computation, this is useful for training. Q , K and V Defined as the first i The rows are vectors q i , k i and v i The matrix.

[0025] Attention Q , K , V ) = softmax

[0026] Multi-head attention

[0027] This article will present a group of ( W Q , W K , W V The matrix is ​​called an attention head, and each layer in a Transformer model has multiple attention heads. While one attention head focuses on the lexical units associated with each lexical unit, by utilizing multiple attention heads, the model can learn for different definitions of "relevance," thus making the relevance encoded by the Transformer understandable to humans. For example, in the context of natural language processing, there might be an attention head that focuses most on the next word for each lexical unit, or an attention head that primarily focuses on the verb and its direct object. Because Transformer models have multiple attention heads, they have the potential to capture relevance relationships at multiple levels and types, from surface to semantics. The multiple outputs of a multi-head attention layer are concatenated to feed forward neural network layers.

[0028] Each encoder consists of two main components: a self-attention mechanism and a feedforward neural network. The self-attention mechanism takes a set of input codes from the previous encoder and measures their correlation with each other to generate a set of output codes. The feedforward neural network then processes each output code individually. These output codes are ultimately passed as input to the next encoder and decoder.

[0029] The first encoder takes positional information and the embedding of the input data as input, rather than encoding itself. The Transformer uses the positional information to leverage the order of the input data, or, in the various examples described herein, the position of objects in the input scene image. In the various examples described herein, positional embeddings can describe the spatial relationships of multiple lexical units relative to other lexical units. For example, input lexical units can represent a 16×16 (or other size grid) superimposed on the image data input frame. Positional embeddings can describe the position of objects / lexical units within the grid (e.g., the position of other lexical units representing other parts of the frame). Therefore, the various techniques described herein are not one-dimensional positional embeddings (such as defining the position of a word in a one-dimensional sentence / paragraph / document in a natural language context), but rather two-dimensional positional embeddings that describe the spatial positions of lexical units within the input data (e.g., two-dimensional positions within a frame, three-dimensional positions within a point cloud, etc.).

[0030] Each decoder layer comprises three components: a self-attention mechanism (e.g., scaled dot product attention), an attention mechanism acting on the encoding (e.g., “encoder-decoder” attention), and a feedforward neural network. The decoder functions similarly to the encoder but incorporates an additional attention mechanism that extracts relevant information from the encoding generated by the encoder. In the self-attention layer, the key, value, and query come from the same place—in the case of the encoder, from the output of previous layers in the encoder. Each position in the encoder can attend to all positions in previous layers of the encoder. In the “encoder-decoder attention” layer (sometimes called “cross-attention”), the query comes from previous decoder layers, and the key and value come from the output of the encoder. This allows each position in the decoder to attend to all positions in the input sequence. The decoder is attending to encoder features.

[0031] In various examples, one or more computing devices (e.g., computing device 120, mobile device 110, etc.) can be used to implement IVA. 0 100. In various examples, implement IVA. 0 One or more computing devices of 100 can be configured to communicate via network 104. Network 104 can be a communication network, such as a local area network (LAN), a wide area network (such as the Internet), or some combination thereof. Implementing IVA 0 One or more computing devices of 100 can communicate with a non-transitory computer-readable storage device 103 (e.g., locally or via a network 104). The non-transitory computer-readable storage device 103 can store instructions that can efficiently perform one or more of the various techniques described herein. For example, these instructions can efficiently implement one or more of the various machine learning models described herein.

[0032] As a generative artificial intelligence task, various methods have been used for video generation. Some methods focus on unconditional generation based on vectors initialized from a predefined probability space (e.g., a Gaussian distribution). Other methods introduce various generation conditions and can be roughly classified as: i) text-to-video generation (T2V): where descriptive text is used as input to guide the generation process; ii) video-to-video generation (V2V): where the structure of the generated video is referenced to video notifications; and iii) image-to-video generation (I2V): which uses a single or a series of images as a basis to generate a continuous sequence of frames. The various techniques described in this paper focus on I2V representation, which provides a clear visual starting point compared to T2V and greater flexibility compared to V2V. However, the method described in this paper achieves layout-based controllability of I2V with a zero-shot mechanism.

[0033] Some methods for controlled video generation focus on encoding images and motion trajectories, primarily for human movement. However, unlike many other methods, the IVA described in this paper... 0 The 100 systems and techniques are not text-based or fine-tuned (or trained) for the task, but rather represent zero-shot image-based generation without any I2V-specific training. Furthermore, the systems and techniques described in this paper can be combined with other language-based layout planning models to generate layout trajectories / sequences based on text (e.g., text cue lexical units), enabling more diverse control conditional inputs.

[0034] like Figure 1 As shown, IVA 0 The input to 100 includes a conditional image 128 (e.g., a user-input image with one or more objects to be animated) and a control layout 130. Additionally, the user can select one or more objects of interest (e.g., using user-drawn and / or pre-existing bounding boxes, segmentation masks, etc.) and can provide a control layout 130 for each object to be animated. The control layout 130 can define the motion path of the objects within the scene provided by the conditional image 128. The user can provide the control layout 130 by providing frame-by-frame bounding boxes (e.g., for two or more different frames of the generated video). In some other examples, the user can use touch input to draw the trajectory of the objects within the scene of the conditional image 128 to show how the objects should move within frames of the resulting video. As previously mentioned, the misaligned motion 132 provided by the control module 124 typically controls how the objects of interest move from one position to another within the user-provided control layout 130. In-place motion 134 ensures the consistency of the objects (e.g., Figure 1The example shows a running horse, while also demonstrating smooth, reasonable pixel-level changes between frames in the resulting video (e.g., the generated video 136), showing the movement of the horse's legs and head as it gallops across the scene.

[0035] Figure 2 Depicting various aspects of this disclosure Figure 1 Controllable Image-to-Video Model (IVA) 0 Additional details of the various components of 100) are provided. As an initial problem, an example representation of latent diffusion-based filling is described, which can be used as the basis for a pre-trained text-to-image model 122 prior to the various modifications described herein. Next, IVA is described. 0 Example instances of 100 are provided, which may include decomposing a controllable I2V task into subtasks that can be potentially propagated using misaligned and in-situ motion modules. Finally, various motion afterimage suppression (MAS) techniques are described to eliminate object afterimage illusions.

[0036] Potential diffusion used to fill

[0037] IVA 0 The goal of IVA is to animate one or more static objects to facilitate their transition from an initial position to a subsequent position based on any user-defined layout (e.g., motion paths or trajectories). In some examples described herein, this task can be viewed as an in-painting task for controlling the movement of such objects, involving: (1) replacing the original object position with a background, while (2) filling the object in the new position based on a user-provided layout. To achieve this goal, IVA... 0 100 is built based on a filler version of latent diffusion (a pre-trained text-to-image model). The latent diffusion model comprises three key components: (1) an autoencoder that maps the image from pixel space to a latent embedding (based on which the diffusion module operates) and projects the embedding back to the pixel space after the denoising step; (2) a text encoder that encodes cues into embeddings for text-to-image conditional constraints; and (3) a U-Net for noise diffusion that iteratively denoises in the latent space guided by timestamps and cue embeddings.

[0038] The inpainting task utilizes a latent diffusion model to modify masked image regions based on given textual conditions. This mask (e.g., mask data) is represented as a 1-channel binary mask and serves as additional input along with the conditional image. The conditional image provides the necessary context for the unpainted portions (e.g., the background) and is processed by the encoder. This is obtained by... To accommodate these additional padding conditions, Diffusion U-Net can incorporate five additional channels into its initial convolutional layers. Given a conditional image... Text prompts and binary mask This fills in the image generated by the model. In the following sections, such as... Figure 2 The description is for IVA 0 100 illustrates and describes the integration of control conditions. Additionally, a technique for handling atomic tasks using this text-to-image filling model is described to achieve controllable I2V (such as IVA). 0 100 instantiations).

[0039] I2V under zero-sample layout constraints

[0040] In IVA 0 In version 100, a controllable image-to-video generation model was introduced. This model utilizes user-provided spatiotemporal object layouts, such as... Figure 2 As shown in column 202 on the left. Given an initial frame. As a conditional image Users can animate a specific object by providing its trajectory (e.g., the motion path of a subsequent frame in the output video). This trajectory can be represented as a sequence of bounding boxes. ,in t Indicates the number of frames. Each frame It can be a 4-dimensional vector, indicating the coordinates of the top-left and bottom-right corners of the bounding box (e.g., in the coordinate space of an image data frame, such as...). x , y (Axis pixel coordinates). However, bounding boxes can also be represented in other ways. Furthermore, segmentation masks can be used to identify objects and their trajectories at the per-pixel level. For simplicity, only the animation of a single object is described. However, it should be noted that IVA... 0 100 is flexible and, when provided with a corresponding layout, can be extended to any number of objects (e.g., multiple objects simultaneously). The detailed model flow is explained below:

[0041] The layout conditions are achieved through spatiotemporal masking:

[0042] Based on the initial frame, via a text-to-image latent diffusion model. Use layout boxes to generate future frames When an object moves from its position Transition to At this time, the expected object will move to the desired position with a smooth interpolated motion, and the appearance of the foreground object and the background context will be consistent. This results in filling in both areas. :(1) From Delete the object in the original region (i.e., in (1) Replace the object with the background, (2) Add the object to the frame New areas in Therefore, as shown in column 202 on the left, by simultaneously masking the starting region... and target area Both are used to generate a padding mask for each frame. As shown in column 202 on the left, from the spatiotemporal layout sequence... In this process, we construct a spatiotemporal masking sequence. Each of them .

[0043] Displacement motion generation:

[0044] A key task of our model is to perform calculations given a spatiotemporal mask. This involves generating misaligned motion under certain conditions. This can be described as generation under layout constraints, requiring the generation of images to adhere to layout conditions (e.g., following user-provided object trajectories). In various examples, it can be like... Figure 3 The model shown uses gated self-attention to generate a layout-to-image generation model. This model encodes object bounding box coordinates as anchor terms. (grounding token) And by adding additional gated self-attention before each cross-attention layer in the text-to-image model, grounding information is fused with visual lexical units. Specifically, such as Figure 2 The middle column 204 and Figure 3 As shown, a (pre-trainable) gated self-attention layer can be inserted into each Transformer block of the text-to-image model 122 (e.g., for filling potential diffusion) as a control module 124. Figure 3 An example of this modification is illustrated. The gated attention-assisted vision Transformer focuses on the target region (e.g., the region identified by anchor words, which in turn represent the input layout) while suppressing feature activations in irrelevant regions. In other words, the gated self-attention layer enables the Transformer module to better understand the spatial locality of the input image patches (identified by anchor words).

[0045] Then, the control module 124 utilizes the appearance of the encapsulated object and the anchoring terms of its frame coordinates to enable precise placement of the object in the output frame. x i The desired location within the image. In various examples, this process can be simplified by using the same Contrastive Language Image Pre-trained (CLIP) neural network model as the image encoder to extract region image features of the cropped object. The bounding box coordinates are then converted using a Fourier transform function (or a similar function). Projected into a continuous embedding, thus controlling spatial position. Therefore, for time... Frames at the location, layout words These conditions are integrated via a linear projection layer. Then, these lexical units are subjected to gated self-attention (e.g., ...). Figure 3 (As shown) interacts with visual lexical units within the same frame, thereby ensuring accurate and context-sensitive misaligned motion generation, so that:

[0046]

[0047]

[0048] Please note that the linking operation above is performed on an attention block.

[0049] Stationary movement animation:

[0050] In various examples, relying solely on misalignment-filling strategies (e.g., using control module 124) only produces basic "copy-paste" animations of objects, resulting in noticeable inconsistencies in object motion across frames of the output video. To achieve smoother and more realistic object movement and ensure consistent visual coherence, in-place motion animation modules (e.g., motion module 126) can be used. Different inter-frame attention mechanisms can be used to aid in achieving this goal. However, typical approaches consistently require large-scale pre-training from video data. Therefore, in a zero-shot context, the pre-trained motion module 126 can be used as temporal attention layers, as shown in middle column 204, which are incorporated into a pre-trained latent diffusion model whose weights are copied from the original text-to-video generation task but are applied in IVA. 0 It is used in 100 controllable I2V tasks.

[0051] Figure 4 An example architecture of motion module 126 according to various aspects of this disclosure is depicted. Animating personalized image models often requires additional tuning to the corresponding video set, which makes it more difficult. However, given a personalized T2I model (e.g., DreamBooth, LoRA, and / or another potentially diffused T2I model), the model can be transformed into an animation generator with little or no training cost while preserving its original domain knowledge and quality. For example, suppose the T2I model is personalized for a specific 2D animation style. In this case, the corresponding animation generator should be able to generate animation clips of that style with appropriate motion, such as foreground / background segmentation, character body movement, etc. This is the task of motion module 126.

[0052] To achieve this goal, one approach is to add a time-aware structure to the T2I model and learn reasonable motion priors from a large-scale video dataset. However, collecting sufficient personalized videos is expensive and time-consuming for the personalization domain. Furthermore, limited data may lead to knowledge loss from the source domain. Therefore, the motion module 126 can be pre-trained separately and / or taken from a similar motion module 126 pre-trained for a given task. This model can then be embedded into the IVA at inference time. 0 100. By doing so, IVA 0 100 avoids specific tuning for each individualized model. The motion module 126 can retain its pre-trained weights. Another advantage of this approach is that once the motion module 126 is trained, it can be inserted into systems including IVA without requiring specialized tuning. 0 In any personalized latent diffusion model of 100, the personalization process hardly changes the feature space of the underlying T2I model.

[0053] like Figure 4 As shown, the motion module can be inserted between pre-trained image layers of the underlying latent diffusion model. When data batches are passed through the image layers and the motion module, their time and spatial axes can be shaped to batch axes (e.g., using shaping operation 402 of equation (3) below).

[0054] The motion module 126 achieves better temporal consistency between object appearance and motion via cross-frame self-attention. Specifically, given visual features of consecutive frames... ,in The feature axis can be shaped, and self-attention can be applied to the time dimension, where , and These refer to the width, height, and channels of the feature map, respectively. For example... Figure 4 As shown on the right, motion module 126 can be instantiated as a time-varying Transformer with a zero-initialized output projection layer. Motion module 126 can take... batch × aisle × frame × high × width The shape's 5-dimensional tensor is used as input.

[0055]

[0056] For the network design of motion module 126, the goal could be efficient information exchange across frames. To achieve this, a temporal Transformer architecture can be used to model motion priors. A temporal Transformer consists of two or more self-attention blocks operating along the time axis. As the feature map is passed through motion module 126... z Spatial Dimensions high and Width Spend It will first be shaped into batch dimensions, resulting in a length of frame of Batch × Height × Width The sequence. Then, the shaped feature map will be projected and passed through several self-attention blocks, namely:

[0057]

[0058] in Q = W Q Z K = W K Z And V = W V Z These are three projections of the shaped feature map. This operation enables motion module 126 to capture the temporal dependencies between features at the same location on the timeline. To expand the receptive field of motion module 126, it can be inserted into each resolution level of the U-shaped diffusion network (similar to control module 124). Additionally, in some examples, sinusoidal position encoding can be added to the self-attention module to enable the network to perceive the temporal position of the current frame in the animation clip.

[0059] The training process for motion module 126 can be similar to that of a latent diffusion model. First, the sampled video can be processed frame-by-frame via a pre-trained autoencoder. Data Encoding to Latent Code Then, the defined forward diffusion scheme can be used to add noise to the underlying code: The diffusion network enhanced with motion module 126 takes noisy latent codes and corresponding text prompts as input and predicts the noise intensity added to the latent codes, encouraged by the L2 loss term. The final training objective of motion module 126 can be:

[0060]

[0061] Please note that a pre-trained version of motion module 126 may be available, allowing a version of motion module 126 to be obtained and inserted into IVA without any labeled video data. 0 100.

[0062] Both the control module 124 and the motion module 126 can be pre-trained on different task-specific data. Note that... Figure 2 A single modified Transformer block with potential diffusion is shown, which has been modified to incorporate control module 124 and motion module 126. Control module 124 and motion module 126 integrate their original image generation and text-to-video tasks under layout condition constraints. These pre-established foundations can be incorporated into IVA. 0 100, to avoid further retraining. Therefore, IVA 0 100 allows for controlled animation of objects in an image without any specific fine-tuning of the I2V.

[0063] Motion afterimage suppression

[0064] Due to IVA 0 The 100 model can be built on top of a text-to-image filling latent diffusion model, so the model can initially fill text cues with a fixed background (such as...). Figure 2 As shown in the middle column 204, the background is prompted to be generated as "Background, goodquality". However, this method sometimes results in afterimages (e.g., a ghostly residual image of an object at its previous position after it has moved). It has been determined that this problem may be related to motion module 126. This is because most current motion modules 126 only generate a limited range of in-situ motion. Therefore, when an object moves from its original position... When there is significant movement, the temporal attention mechanism fails to maintain appearance consistency. Conversely, temporal attention often incorrectly focuses on... In An afterimage is produced at that location.

[0065] To suppress such motion afterimages, additional anchoring terms can be used in background generation. For example... Figure 2 As shown, in the middle column 204, anchor terms that encode both the bounding box and the word 'background' (via CLIP text embedding) can be used. In this case, the control module 124 is forced to [implement certain controls] in the frame. In Background generation is performed on small objects. This method successfully avoids afterimage illusions, but may still have difficulties with large areas / large objects. This limitation may stem from the training of the control module 124 in background reconstruction, which may not be able to handle the filling of large regions with a single lexical. To overcome this problem, the right sidebar 206 shows the motion afterimage suppression (MAS) technique. The motion afterimage suppression technique can combine two background generation methods based on object size and intersection-over-union (IoU). In various examples, objects can first be classified by size (small, medium, large) according to a predefined area threshold. Small objects can use additional anchor lexicals for background filling, while large objects can be left unfilled. For medium-sized objects, such as... Figure 2 As shown in column 206 on the right, we can first calculate... and IoU between .if —This instruction and The overlap between them—can be addressed by iteratively removing non-overlapping background areas (e.g., Zhongyu The model divides non-overlapping bounded regions into a grid. Then, each grid is filled with background-class lexics. MAS can significantly mitigate the afterimage object illusion problem while maintaining high-quality background generation.

[0066] Potential diffusion model

[0067] As described in this paper, the basic text-to-image model 122 can be a pre-trained latent diffusion model, which can be modified as previously described. Generally, latent diffusion models are generative models that iteratively denoise normally distributed variables by reversing a fixed-length Markov forward process to learn the data distribution. In some cases, denoising can be performed in the latent space instead of the pixel space, which is computationally efficient because it reduces the dimensionality of the image. Furthermore, the use of the latent space omits high-frequency noise within a given image. An example of a latent diffusion model is stable diffusion, which comprises three main components: a variational autoencoder (VAE) for transforming a given input into the latent space; a text encoder for processing a given text, with image generation conditioned on this given text; and a temporally constrained UNet for predicting noise added to the image latent space, which is conditionally constrained by the text embedding. Mathematically, a conditionally constrained latent diffusion model can be learned by optimizing the following loss:

[0068]

[0069] in, Input provided by VAE The potential version, namely It is input x At time step The version with added noise, where ,and Decreasing with timestamp Noise express. It's UNet. Finally, This represents the conditional constraint variables, and for text-guided models, it is given by processing the given text using the CLIP text encoder.

[0070] Figure 5 This is a block diagram illustrating an example architecture 500 of a computing device according to various aspects of this disclosure, which can be used to instantiate various machine learning models, such as latent diffusion models, generative models, Transformers, encoders, and / or other models described herein. It should be noted that not all devices will include all components of architecture 500, and some user devices may include additional components not shown in architecture 500. Architecture 500 may include one or more processing elements 504 for executing instructions and retrieving data stored in storage unit 502. Processing element 504 may include at least one processor. Any suitable processor may be used. For example, processing element 504 may include one or more digital signal processors (DSPs). Storage element 502 may include one or more different types of memory, data storage, or computer-readable storage media dedicated to different purposes within architecture 500. For example, storage element 502 may include flash memory, random access memory, disk storage, etc. Different portions of storage element 502 may, for example, be used for storing program instructions, images, or other digital works to be executed by processing element 504, and / or for removable storage for transferring data to other devices, etc. In addition, storage element 502 can also store parameters and / or machine learning models generated using the various techniques described herein.

[0071] Storage element 502 may also store software for execution by processing element 504. Operating system 522 provides a user interface for operating the computing device and facilitates communication and commands between applications running on architecture 500 and their various hardware components. Transmission application 524 may be configured to receive images, audio, and / or video from another device (e.g., a mobile device, an image capture device, and / or a display device) or from image sensor 532 and / or microphone 570 included in architecture 500.

[0072] When implemented in some user device, architecture 500 may also include a display component 506. Display component 506 may include one or more light-emitting diodes (LEDs) or other suitable display lights. Furthermore, in some examples, display component 506 may include, for example, one or more devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), gas plasma-based flat panel displays, LCD projectors, raster projectors, infrared projectors, or other types of display devices. As described herein, display component 506 can effectively display suggested personalized search queries generated according to the various techniques described herein.

[0073] Architecture 500 may also include one or more input devices 508 operable to receive input from a user. Input devices 508 may include, for example, buttons, touchpads, touchscreens, scroll wheels, joysticks, keyboards, mice, trackballs, keypads, light guns, game controllers, or any other such devices or elements through which the user provides input to architecture 500. These input devices 508 may be incorporated into architecture 500 or operably coupled to architecture 500 via wired or wireless interfaces. In some examples, architecture 500 may include a microphone 570 or a microphone array for capturing sound such as a voice request. In various examples, the audio captured by microphone 570 may be streamed to an external computing device via communication interface 512.

[0074] When the display component 506 includes a touch-sensitive display, the input device 508 may include a touch sensor that operates in conjunction with the display component 506 to allow a user to interact with an image displayed by the display component 506 using touch input (e.g., with a finger or stylus). The architecture 500 may also include a power source 514, such as a wired AC converter or a rechargeable battery operable to be charged via conventional plug-in methods or other methods such as capacitive or inductive charging.

[0075] Communication interface 512 may include one or more wired or wireless components operable to communicate with one or more other computing devices. For example, communication interface 512 may include wireless communication module 536 configured to communicate on a network such as network 604 according to any suitable wireless protocol such as IEEE 802.11 or other suitable wireless local area network (WLAN) protocol. Short-range interface 534 may be configured to communicate using one or more short-range wireless protocols such as Near Field Communication (NFC), Bluetooth, Bluetooth LE, etc. Mobile interface 540 may be configured to communicate using cellular or other mobile protocols. Global Positioning System (GPS) interface 538 may communicate with one or more Earth-orbiting satellites or other suitable text identification systems to identify the location of architecture 500. Wired communication module 542 may be configured to communicate according to the USB protocol or any other suitable protocol.

[0076] Architecture 500 may also include one or more sensors 530, such as one or more position sensors, image sensors and / or motion sensors. Figure 5 Image sensor 532 is shown. Some examples of architecture 500 may include multiple image sensors 532. For example, a panoramic camera system may include multiple image sensors 532 to generate multiple images and / or video frames that can be stitched together and blended to form a seamless panoramic output. Examples of image sensors 532 may be cameras configured to capture color information, image geometry information, and / or ambient light information. In some other examples, image sensor 532 may include a depth sensor and / or multiple depth sensors. For example, image sensor 532 may include a TOF sensor, a stereo depth sensor, a lidar sensor, radar, etc.

[0077] As described above, multiple devices can be employed in a single system. In such a multi-device system, each device may include different components for performing different aspects of system processing. The multiple devices may include overlapping components. Components of a computing device, as described herein, are exemplary and may be positioned as independent devices or may be included, wholly or partially, as components of a larger device or system.

[0078] The example system used for sending and providing data will now be described in detail. Specifically, Figure 6 An example computing environment is shown in which the embodiments described herein can be implemented. For example, Figure 6 The computing environment can be used to provide various machine learning models described herein as services over a network, wherein one or more of the techniques described herein can be requested by a first computing device and can be executed by different computing devices configured to communicate with the first computing device over a network. Figure 6 This diagram schematically illustrates an example of a data center 65 that can provide computing resources to users 60a and 60b (which may be referred to herein as users 60 in the singular or plural) via network 604 through user computers 62a and 62b (which may be referred to herein as user computers 62 in the singular or plural) . Data center 65 can be configured to provide computing resources for executing applications permanently or on demand. The computing resources provided by data center 65 can include various types of resources, such as gateway resources, load balancing resources, routing resources, networking resources, computing resources, volatile and non-volatile memory resources, content distribution resources, data processing resources, data storage resources, data communication resources, etc. Each type of computing resource is available in multiple specific configurations. For example, data processing resources may be available as virtual machine instances that can be configured to provide various web services. Furthermore, combinations of resources may be available via the network and may be configured as one or more web services. These instances can be configured to execute applications, including web services such as application services, media services, database services, processing services, gateway services, storage services, routing services, security services, encryption services, load balancing services, and more. In various examples, instances can be configured to execute one or more of the various machine learning techniques described herein.

[0079] These services can be configured using set-up or custom applications and can be configured in terms of size, execution, cost, latency, type, duration, accessibility, and any other dimensions. These web services can be configured as infrastructure available to one or more clients and can include one or more applications of systems or software configured as one or more clients. These web services can be made available via one or more communication protocols. These communication protocols can include, for example, Hypertext Transfer Protocol (HTTP) or non-HTTP protocols. These communication protocols can also include, for example, more reliable transport layer protocols such as Transmission Control Protocol (TCP), and less reliable transport layer protocols such as User Datagram Protocol (UDP). Data storage resources can include file storage devices, block storage devices, etc.

[0080] The computing resources available for each type or configuration can vary in size, such as large resources—consisting of many processors, a large amount of memory, and / or a large storage capacity—and small resources—consisting of fewer processors, a smaller amount of memory, and / or a smaller storage capacity. For example, a customer might choose to allocate multiple small processing resources as a web server, and / or allocate one large processing resource as a database server.

[0081] Data center 65 may include servers 66a and 66b (referred to herein as servers 66 in the singular and plural) providing computing resources. These resources may be available as bare metal resources or as virtual machine instances 68a-d (referred to herein as virtual machine instances 68 in the singular and plural). At least in some examples, server manager 67 may control the operation and / or maintenance of server 66. Virtual machine instances 68c and 68d are reversion switching virtual machines (“RSVM”) instances. RSVM virtual machine instances 68c and 68d may be configured to perform all or any part of the techniques for improving reversion switching as described above and in detail herein, and / or any other disclosed techniques. It should be understood that, although Figure 6 The specific example shown includes one RSVM virtual machine in each server, but this is just an example. A server may contain more than one RSVM virtual machine, or it may not contain any RSVM virtual machines.

[0082] The availability of virtualization technologies for computing hardware has provided the benefits of offering customers large-scale computing resources and allowing these resources to be shared efficiently and securely among multiple customers. For example, virtualization technologies can allow the sharing of physical computing devices among multiple users by providing each user with one or more virtual machine instances hosted by the physical computing device. A virtual machine instance can be a software emulation of a specific physical computing system that acts as a disparate logical computing system. Such virtual machine instances provide isolation among multiple operating systems sharing a given physical computing resource. Furthermore, some virtualization technologies can provide virtual resources spanning one or more physical resources, such as a single virtual machine instance with multiple virtual processors spanning multiple disparate physical computing systems.

[0083] refer to Figure 6 Network 604 may be, for example, a publicly accessible network of linked networks and may be operated by various disparate parties such as the Internet. In other embodiments, network 604 may be a private network, such as a corporate or university network that is completely or partially inaccessible to non-privileged users. In yet another embodiment, network 604 may include one or more private networks that are accessible and / or originate from the Internet.

[0084] Network 604 can provide access to user computer 62. User computer 62 can be a computer used by user 60 or other clients of data center 65. For example, user computer 62a or 62b can be a server, desktop or laptop computer, tablet computer, wireless phone, personal digital assistant (PDA), e-book reader, game console, set-top box, or any other computing device capable of accessing data center 65. User computer 62a or 62b can be directly connected to the Internet (e.g., via cable modem or digital subscriber line (DSL)). Although only two user computers 62a and 62b are depicted, it should be understood that multiple user computers are possible.

[0085] User computer 62 can also be used to configure various aspects of the computing resources provided by data center 65. In this regard, data center 65 can provide a gateway or web interface through which aspects of its operation can be configured using a web browser application running on user computer 62. Alternatively, a standalone application running on user computer 62 can access application programming interfaces (APIs) exposed by data center 65 to perform configuration operations. Other mechanisms for configuring the operation of various web services available at data center 65 can also be utilized.

[0086] Figure 6 The server 66 shown may be a server appropriately configured to provide the aforementioned computing resources, and may provide computing resources for executing one or more web services and / or applications. In one embodiment, the computing resources may be virtual machine instances 68. In the example of virtual machine instances, each of the servers 66 may be configured to execute an instance manager 63a or 63b capable of executing virtual machine instances 68 (hereinafter referred to as instance manager 63 in the singular or plural). For example, instance manager 63 may be a virtual machine monitor (VMM) or another type of program configured to execute virtual machine instances 68 on the server 66. As described above, each of the virtual machine instances 68 may be configured to execute all or part of an application.

[0087] It should be understood that although the embodiments disclosed above discuss the context of virtual machine instances, other types of implementations can be utilized in conjunction with the concepts and techniques disclosed herein. For example, the embodiments disclosed herein can also be utilized in conjunction with computing systems that do not utilize virtual machine instances.

[0088] exist Figure 6In the example data center 65 shown, router 61 can be used to interconnect servers 66a and 66b. Router 61 can also be connected to gateway 64, which is connected to network 604. Router 61 can be connected to one or more load balancers and can manage communication within the network in data center 65, individually or in combination, for example, by appropriately forwarding packets or other data communications based on the characteristics of such communication (e.g., header information including source and / or destination addresses, protocol identifiers, size, processing requirements, etc.) and / or the characteristics of private networks (e.g., routing based on network topology, etc.). It should be understood that, for simplicity, various aspects of the computing system and other apparatus of this example are shown without showing certain general details. In other embodiments, additional computing systems and other apparatus may be interconnected, and may be interconnected in different ways.

[0089] exist Figure 6 In the example data center 65 shown, data center 65 is also used, at least in part, to route various communications to, from, and / or between servers 66a and 66b. Although Figure 6 A router 61 is depicted positioned between gateway 64 and data center 65, but this is merely an exemplary configuration. For example, in some cases, data center 65 may be positioned between gateway 64 and router 61. In some cases, data center 65 may examine portions of incoming communication from user computer 62 to determine one or more appropriate servers 66 to receive and / or process the incoming communication. Data center 65 may determine the appropriate server for receiving and / or processing incoming communication based on factors such as the identity, location, or other attributes associated with user computer 62, the nature of the communication and the task associated with it, the priority of the communication and the task associated with it, the duration of the communication and the task associated with it, the size of the communication and the task associated with it, and / or estimated resource usage, among many other factors. Data center 65 may, for example, collect or otherwise access status information and other information associated with various tasks to, for example, assist in managing communications and other operations associated with such tasks.

[0090] It should be understood that Figure 6 The network topologies shown have been greatly simplified, and more networks and networking devices can be used to interconnect the various computing systems disclosed herein. These network topologies and devices should be obvious to those skilled in the art.

[0091] It should also be understood that Figure 6The data center 65 described herein is merely illustrative and may be implemented in other ways. It should also be understood that servers, gateways, or other computing devices may include any combination of hardware or software capable of interacting with and performing functions of the type described herein, including, but not limited to: desktop or other computers, database servers, network storage devices and other network devices, PDAs, tablets, mobile phones, cordless phones, pagers, electronic notebooks, internet applications, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products including appropriate communication capabilities.

[0092] A network established by an entity such as a corporation or public sector organization can be referred to as a provider network, which provides one or more web services (such as various types of cloud-based computing or storage) accessible via the Internet and / or other networks to a distributed set of clients. Such provider networks may include multiple data centers hosting a collection of various resource pools such as physical and / or virtualized computer servers, storage devices, networking devices, etc., for implementing and distributing the infrastructure and web services provided by the provider network. In some embodiments, resources may be provided to clients in various units associated with web services—such as storage capacity for storage, processing power for processing, as instances, as collections of related services, etc. For example, a virtual computing instance may include one or more servers with specified computing power (which can be specified by indicating the type and number of CPUs, main memory size, etc.) and a specified software stack (e.g., a specific version of an operating system, which can then run on top of a hypervisor).

[0093] In different embodiments, various types of computing devices, such as computer servers, storage devices, and network devices, can be used individually or in combination to implement the resources of the provider network. In some embodiments, direct access to resource instances can be provided to clients or users, for example, by providing users with administrator login and passwords. In other embodiments, the provider network operator can allow clients to specify execution requirements for designated client applications and schedule the execution of the application on an execution system suitable for the application (such as an application server instance, a Java Virtual Machine (JVM), a general-purpose or special-purpose operating system or high-performance computing system supporting various interpreted or compiled programming languages ​​(such as Ruby, Perl, Python, C, C++, etc.)) without requiring, for example, direct client access to the instance or execution system. In some implementations, a given execution system can utilize one or more resource instances; in other implementations, multiple execution systems can be mapped to a single resource instance.

[0094] In many environments, operators of provider networks that implement different types of virtualized compute, storage, and / or other network-accessible capabilities can allow customers to reserve or purchase access to resources using various resource acquisition models. Compute resource providers can provide customers with facilities to select and initiate desired compute resources, deploy application components to compute resources, and maintain applications running in the environment. Furthermore, when application demands or capacity requirements change, compute resource providers can provide customers with additional facilities to quickly and easily scale up or down the quantity and type of resources allocated to applications, manually or through autoscaling. Compute resources provided by compute resource providers can be available in discrete units, which may be referred to as instances. An instance can represent a physical server hardware system, a virtual machine instance running on a server, or some combination of both. Various types and configurations of instances can be made available, including resources of different sizes running different operating systems (OS) and / or hypervisors, as well as various installed software applications, runtimes, etc. Instances can be further available in specific availability zones, such as logical areas, fault-tolerant zones, data centers, or other geographical locations representing the underlying compute hardware. Instances can be replicated within or across availability zones to improve redundancy, and instances can be migrated within or across specific availability zones. As an example, the latency for a client communicating with a specific server within an availability zone can be lower than the latency for a client communicating with different servers. This allows instances to be migrated from higher-latency servers to lower-latency servers, improving the overall client experience.

[0095] In some embodiments, the provider network can be organized into multiple geographic regions, and each region can include one or more availability zones. An availability zone (also referred to as an availability container) can then include one or more disparate locations or data centers, configured in such a way that resources in a given availability zone can be isolated from or prevented from failing in other availability zones. That is, a failure in one availability zone may not be expected to cause failures in any other availability zone. Therefore, the availability profile of a resource instance is designed to be independent of the availability profiles of resource instances in different availability zones. Clients can be able to protect their applications from failures at a single location by launching multiple application instances in the respective availability zones. Meanwhile, in some implementations, inexpensive and low-latency network connectivity can be provided between resource instances residing within the same geographic region (and network transmission between resources in the same availability zone can even be faster).

[0096] Figure 7 This is a block diagram illustrating an example process for controllable image-to-video generation according to embodiments of the present disclosure. For clarity and brevity, descriptions previously referenced herein will not be repeated. Figures 1 to 6Described Figure 7 Those actions in the text. Figure 7 The actions of the process depicted in the flowchart can represent a series of instructions comprising computer-readable machine code executable by one or more processing units of one or more computing devices. In various examples, the computer-readable machine code may include instructions selected from the native instruction set and / or operating system (or multiple operating systems) of one or more computing devices. Although the accompanying drawings and discussion show certain operational steps of the system in a particular order, the described steps may be performed in a different order (and some steps may be removed or added) without departing from the intent of this disclosure.

[0097] Process 700 may begin at action 710, where a first frame of image data representing at least a first object may be received. The first frame of image data may be a subject image with one or more objects (e.g., the first object) that the user wants to animate within the scene represented by the first frame of image data.

[0098] Processing can continue at action 720, where first input data, including the selection of a first object in the first frame, can be received. This first input data can identify an object (e.g., a first object) that the user wants to animate within the scene represented by the first frame of image data. This first input data can be provided by the user drawing a bounding box around the object, or by the user selecting an object previously identified using semantic segmentation and / or a computer vision-based object detector.

[0099] Processing can continue at action 730, where second input data can be received, including at least a first bounding box indicating the target position of the first object in a second frame of the video animates the first object. For example, the user can provide a layout including a bounding box representing the position within the first frame where the selected object should be moved in the video animates the selected object. In some examples, instead of providing bounding box positions representing the movement of objects in the video, the user can draw a motion path and / or trajectory on the first frame of the image data, indicating the path the first object should take in the video. In some examples, bounding boxes for different object positions can be automatically generated along the user-provided motion path and / or trajectory, allowing the animated object to be animated in IVA. 0 The 100 video was filled in those positions during the video generation process.

[0100] Processing can continue at action 740, where the latent diffusion text-to-image model (e.g., modified as described herein) can generate a first plurality of visual lexical units (e.g., intermediate visual lexical units). For example, the autoencoder and / or self-attention block of the latent diffusion model can generate the first plurality of visual lexical units representing pixels of the input image (e.g., representing both the first object and the background of the first frame of image data).

[0101] Processing can continue at action 750, where one or more first anchor terms representing the position of the first bounding box can be generated. Anchor terms can represent the above references. Figure 2 The description describes the coordinates of various bounding boxes in a spatiotemporal mask, which represents the original position of the selected object and the updated position of the selected object (defined by a user-provided layout) frame by frame. For example, anchor terms may represent a first bounding box in the first frame of image data (e.g., the position of the object in the initial frame) and / or a user-provided layout (e.g., the target position of the selected object along a motion path). Anchor terms may be fused with a first plurality of visual terms, allowing the latent diffusion model to focus on the target region while suppressing feature activations in irrelevant regions (e.g., using gated attention). Therefore, at action 760, first intermediate data (e.g., fused visual and anchor terms) can be generated by combining the first plurality of visual terms with one or more first anchor terms using at least one gated attention layer.

[0102] Processing can continue at action 770, where second intermediate data can be generated by inputting the first intermediate data into at least a first temporal attention layer. For example, after generating the first intermediate data using control module 124, temporal attention (e.g., via motion module 126) can be applied to perform the above-mentioned reference. Figure 4 The described shaping operation produces a second intermediate data.

[0103] Processing can continue at action 780, where the latent diffusion text-to-image model can generate a video animates the first object based on second intermediate data, the video comprising at least a first frame and a second frame. For example, after the latent diffusion text-to-image model's denoising process, the decoder can decode the latent representation of each frame into pixel space to generate the frames of the video. The latent diffusion text-to-image model can fill in the selected object at locations defined by a user-provided layout (e.g., a first bounding box indicating the target location, and any other target locations defined by the user-provided layout). Additionally, the latent diffusion text-to-image model can fill in the object's previously located location (e.g., the object's position in the first frame) with seemingly believable background image data that matches the scene in the first frame of the image data. Thus, the video can animate the first object moving within the scene according to a user-defined layout. The temporal attention mechanism of motion module 126 can provide smooth pixel-level changes, making the object appear to move realistically between frames of the video (e.g., depicting typical arm, leg, and body movements of a running person).

[0104] Terms and Conditions

[0105] Various embodiments of this disclosure may be described in view of the following terms:

[0106] Clause 1. A computer-implemented method comprising:

[0107] Receive a first image representing at least one first object;

[0108] Receive first input data, the first input data including a selection of a first object in the first image, the selection of the first object instructing the first object to be animated along a motion path;

[0109] Receive second input data, the second input data including at least a first bounding box indicating the target location of the first object;

[0110] The first frame of image data is input into a pre-trained latent diffusion text-to-image model;

[0111] The first plurality of visual lexical units are generated using the pre-trained latent diffusion text-to-image model and the first image;

[0112] Generate one or more first anchor words representing the position of the first bounding box;

[0113] A first intermediate visual word is generated by combining one or more first visual words with the first plurality of anchor words using a gated attention layer.

[0114] The second intermediate visual lexical is generated by combining the first intermediate visual lexical with the text cue lexical using a cross-attention layer.

[0115] Output visual lexical units are generated by inputting the second intermediate visual lexical unit into at least one temporal attention layer; and

[0116] The video frame is generated by denoising the output visual lexical units using a pre-trained latent diffusion text-to-image model, wherein the video frame includes a representation of the first object at the target location.

[0117] Clause 2. The computer-implemented method as described in Clause 1 further includes:

[0118] Determine a second bounding box representing the position of a first object in a first image;

[0119] Determine that a first region associated with a second bounding box in the first image and a second region associated with a first bounding box in the frame, in a coordinate space shared by the first image and the frame, at least partially overlaps;

[0120] Determine the bounded region within the second bounding box that does not overlap with the first bounding box; and

[0121] Generate one or more second anchor words that indicate the bounded region is background, wherein the one or more second anchor words are combined with a first intermediate visual word using a cross-attention layer.

[0122] Clause 3. The computer-implemented method as described in Clause 1, wherein the pre-trained latent diffusion text-to-image model is a zero-sample image-to-video model for generating video.

[0123] Clause 4. The computer-implemented method as described in Clause 1, wherein portions of the first image other than the first object are maintained within frames of a video.

[0124] Clause 5. A method comprising:

[0125] Receive a first image representing at least one first object;

[0126] Receive first input data, the first input data including a selection of the first object in the first image for animation;

[0127] Receive second input data, the second input data including at least a first bounding box indicating the target location of the first object;

[0128] Using a latent diffuse text-to-image model and the first image, a first plurality of visual lexical units are generated;

[0129] Generate one or more first anchor terms representing the position of the first bounding box; and

[0130] The potential diffuse text-to-image model generates a video that animates the first object based on the first plurality of visual lexical units and the one or more first anchoring lexical units, wherein the video includes at least the first image and a second image that includes a representation of the first object at the target location.

[0131] Clause 6. The method described in Clause 5 further includes:

[0132] The encoder of the latent diffuse text-to-image model is used to generate the first plurality of visual lexical units representing the first image; and

[0133] First intermediate data is generated by combining the first plurality of visual lexical units with the one or more first anchor lexical units using a gated attention layer.

[0134] Clause 7. The method described in Clause 6 further includes:

[0135] Determine the second plurality of visual lexical units representing the second image; and

[0136] A modified second plurality of visual lexical units is generated by applying self-attention between the first plurality of visual lexical units and the second plurality of visual lexical units.

[0137] Clause 8. The method described in Clause 7 further includes:

[0138] A latent diffusion text-to-image model is used to denoise the modified second plurality of visual lexical units, thereby transforming the modified second plurality of visual lexical units from the latent space to the pixel space, thus generating a second image of the video.

[0139] Clause 9. The method described in Clause 5 further includes:

[0140] Determine a second bounding box representing the position of a first object in a first image;

[0141] It is determined that the first bounding box and the second bounding box at least partially overlap in the coordinate space of the first image and the second image;

[0142] Determine the region within the second bounding box that does not overlap with the first bounding box; and

[0143] Generate one or more second anchor words that indicate that the region is background, wherein the one or more second anchor words are combined with the representation of the first plurality of visual words using a cross-attention layer.

[0144] Clause 10. The method described in Clause 5 further includes:

[0145] Determine a second bounding box representing the position of a first object in the first image; and

[0146] Generate first mask data including the union of the first bounding box and the second bounding box, wherein the potential diffuse text to the image model masks the image data in the first image defined by the first mask data for the filling task.

[0147] Clause 11. The method described in Clause 5 further includes:

[0148] Determine a second bounding box representing the position of a first object in the first image; and

[0149] Generate one or more second anchor words that include text indicating that the second bounding box is the background.

[0150] Clause 12. The method as described in Clause 5, wherein the potential diffusion text-to-image model is a zero-sample image-to-video model.

[0151] Clause 13. A system comprising:

[0152] At least one processor; and

[0153] A non-transitory computer-readable storage memory that stores instructions that, when executed by the at least one processor, are effective for:

[0154] Receive a first image representing at least one first object;

[0155] Receive first input data, the first input data including a selection of the first object in the first image for animation;

[0156] Receive second input data, the second input data including at least a first bounding box indicating the target location of the first object;

[0157] Using a latent diffuse text-to-image model and the first image, a first plurality of visual lexical units are generated;

[0158] Generate one or more first anchor terms representing the position of the first bounding box; and

[0159] The potential diffuse text-to-image model generates a video that animates the first object based on the first plurality of visual lexical units and the one or more first anchoring lexical units, wherein the video includes at least the first image and a second image that includes a representation of the first object at the target location.

[0160] Clause 14. In a system as described in Clause 13, the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for:

[0161] The encoder of the latent diffuse text-to-image model is used to generate the first plurality of visual lexical units representing the first image; and

[0162] First intermediate data is generated by combining the first plurality of visual lexical units with the one or more first anchor lexical units using a gated attention layer.

[0163] Clause 15. The system as described in Clause 14 includes: a non-transitory computer-readable storage memory that stores instructions which, when executed by the at least one processor, are effective for:

[0164] Determine the second plurality of visual lexical units representing the second image; and

[0165] A modified second plurality of visual lexical units is generated by applying self-attention between the first plurality of visual lexical units and the second plurality of visual lexical units.

[0166] Clause 16. The system as described in Clause 15 includes: a non-transitory computer-readable storage memory that stores instructions which, when executed by the at least one processor, are effective for:

[0167] A latent diffusion text-to-image model is used to denoise the modified second plurality of visual lexical units, thereby transforming the modified second plurality of visual lexical units from the latent space to the pixel space, thus generating a second image of the video.

[0168] Clause 17. In a system as described in Clause 13, the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for:

[0169] Determine a second bounding box representing the position of a first object in a first image;

[0170] It is determined that the first bounding box and the second bounding box at least partially overlap in the coordinate space of the first image and the second image;

[0171] Determine the region within the second bounding box that does not overlap with the first bounding box; and

[0172] Generate one or more second anchor words that indicate that the region is background, wherein the one or more second anchor words are combined with the representation of the first plurality of visual words using a cross-attention layer.

[0173] Clause 18. In a system as described in Clause 13, the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for:

[0174] Determine a second bounding box representing the position of a first object in the first image; and

[0175] Generate first mask data including the union of the first bounding box and the second bounding box, wherein the potential diffuse text to the image model masks the image data in the first image defined by the first mask data for the filling task.

[0176] Clause 19. The system as described in Clause 13, wherein the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for:

[0177] Determine a second bounding box representing the position of a first object in the first image; and

[0178] Generate one or more second anchor words that include text indicating that the second bounding box is the background.

[0179] Clause 20. The method as described in Clause 13, wherein the potential diffusion text-to-image model is a zero-sample image-to-video model.

[0180] While the various systems described herein can be implemented in software or with code executed by the aforementioned general-purpose hardware, alternatively, they can be implemented in dedicated hardware or a combination of software / general-purpose hardware and dedicated hardware. If implemented in dedicated hardware, each can be implemented as a circuit or state machine employing any one or more of a variety of techniques. These techniques may include, but are not limited to, discrete logic circuits with logic gates for implementing various logical functions when one or more data signals are applied, application-specific integrated circuits with appropriate logic gates, or other components. Such techniques are generally well known to those skilled in the art and therefore will not be described in detail herein.

[0181] The flowcharts and methods described in this paper illustrate the functionality and operation of various implementations. If implemented in software, each block or step can represent a module, fragment, or section of code including program instructions to perform a specified logical function. Program instructions can be implemented in the form of source code or machine code; source code consists of human-readable statements written in a programming language, while machine code consists of digital instructions recognizable by a suitable execution system, such as a processing component in a computer system. If implemented in hardware, each block can represent a circuit or multiple interconnected circuits to perform a specified logical function.

[0182] While the flowcharts and methods described herein can depict a specific execution order, it should be understood that the execution order may differ from that described. For example, the execution order of two or more blocks or steps may be shuffled relative to the described order. Furthermore, two or more blocks or steps may be executed simultaneously or partially simultaneously. Additionally, in some embodiments, one or more blocks or steps may be skipped or omitted. It should be understood that all such variations are within the scope of this disclosure.

[0183] Furthermore, any logic or application including software or code described herein can be embodied in any non-transitory computer-readable medium or memory for use by or in connection with an instruction execution system, such as a processing component in a computer system. In this sense, the logic can include, for example, statements comprising instructions and statements that are available from the computer-readable medium and executable by the instruction execution system. In the context of this disclosure, "computer-readable medium" can be any medium capable of containing, storing, or maintaining the logic or application described herein for use by or in connection with an instruction execution system. A computer-readable medium can be any of a number of physical media, such as magnetic media, optical media, or semiconductor media. More specific examples of computer-readable media include, but are not limited to, magnetic magnetic tape, magnetic floppy disk, magnetic hard disk, memory card, solid-state drive, USB flash drive, or optical disk. Furthermore, a computer-readable medium can be random access memory (RAM), including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of storage devices.

[0184] It should be emphasized that the above embodiments of this disclosure are merely possible examples of implementation methods described for the purpose of clearly understanding the principles of this disclosure. Many variations and modifications can be made to the above examples without substantially departing from the spirit and principles of this disclosure. All such modifications and variations are intended to be included within the scope of this disclosure and are protected by the following claims.

Claims

1. A method comprising: Receive a first image representing at least one first object; Receive first input data, the first input data including a selection of the first object in the first image for animation; Receive second input data, the second input data including at least a first bounding box indicating the target location of the first object; Using a latent diffuse text-to-image model and the first image, a first plurality of visual lexical units are generated; Generate one or more first anchor words representing the position of the first bounding box; as well as The latent diffusion text-to-image model generates a video that animates the first object based on the first plurality of visual lexical units and the one or more first anchoring lexical units, wherein the video includes at least the first image and a second image including a representation of the first object at the target location.

2. The method of claim 1, further comprising: The encoder of the latent diffuse text-to-image model is used to generate the first plurality of visual lexical units representing the first image; as well as First intermediate data is generated by combining the first plurality of visual lexical units with one or more first anchored lexical units using a gated attention layer.

3. The method of claim 2, further comprising: Determine a second plurality of visual lexical units representing the second image; as well as A modified second plurality of visual lexical units are generated by applying self-attention between the first plurality of visual lexical units and the second plurality of visual lexical units.

4. The method of claim 3, further comprising: The modified second plurality of visual lexical units are denoised using the latent diffusion text-to-image model to transform the modified second plurality of visual lexical units from the latent space to the pixel space, thereby generating the second image of the video.

5. The method according to any one of claims 1 to 2, further comprising: Determine a second bounding box representing the position of the first object in the first image; It is determined that the first bounding box and the second bounding box at least partially overlap in the coordinate space of the first image and the second image; Determine the region within the second bounding box that does not overlap with the first bounding box; as well as One or more second anchor words are generated to indicate that the region is background, wherein the one or more second anchor words are combined with the representation of the first plurality of visual words using a cross-attention layer.

6. The method of any one of claims 1 to 2, further comprising: Determine a second bounding box representing the position of the first object in the first image; as well as Generate first mask data including the union of the first bounding box and the second bounding box, wherein the latent diffuse text to image model masks image data in the first image defined by the first mask data for a filling task.

7. The method of any one of claims 1 to 2, further comprising: Determine a second bounding box representing the position of the first object in the first image; as well as Generate one or more second anchor words that include text indicating that the second bounding box is the background.

8. The method of any one of claims 1 to 2, wherein the potential diffusion text-to-image model is a zero-sample image-to-video model.

9. A system comprising: At least one processor; and A non-transitory computer-readable storage memory that stores instructions, which, when executed by the at least one processor, are effective for: Receive a first image representing at least one first object; Receive first input data, the first input data including a selection of the first object in the first image for animation; Receive second input data, the second input data including at least a first bounding box indicating the target location of the first object; Using a latent diffuse text-to-image model and the first image, a first plurality of visual lexical units are generated; Generate one or more first anchor words representing the position of the first bounding box; as well as The latent diffusion text-to-image model generates a video that animates the first object based on the first plurality of visual lexical units and the one or more first anchoring lexical units, wherein the video includes at least the first image and a second image including a representation of the first object at the target location.

10. The system of claim 9, wherein the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for: The encoder of the latent diffuse text-to-image model is used to generate a first plurality of visual lexical units representing the first image; and First intermediate data is generated by combining the first plurality of visual lexical units with one or more first anchored lexical units using a gated attention layer.

11. The system of claim 10, wherein the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for: Determine a second plurality of visual lexical units representing the second image; and A modified second plurality of visual lexical units are generated by applying self-attention between the first plurality of visual lexical units and the second plurality of visual lexical units.

12. The system of claim 11, wherein the non-transitory computer-readable storage memory stores instructions that, when executed by the at least one processor, are effective for: The modified second plurality of visual lexical units are denoised using the latent diffusion text-to-image model to transform the modified second plurality of visual lexical units from the latent space to the pixel space, thereby generating the second image of the video.

13. The system of any one of claims 9 to 10, wherein the non-transitory computer-readable storage memory stores instructions, which, when executed by the at least one processor, are effective for: Determine a second bounding box representing the position of the first object in the first image; It is determined that the first bounding box and the second bounding box at least partially overlap in the coordinate space of the first image and the second image; Determine the region within the second bounding box that does not overlap with the first bounding box; as well as One or more second anchor words are generated to indicate that the region is background, wherein the one or more second anchor words are combined with the representation of the first plurality of visual words using a cross-attention layer.

14. The system of any one of claims 9 to 10, wherein the non-transitory computer-readable storage memory stores instructions, which, when executed by the at least one processor, are effective for: Determine a second bounding box representing the position of the first object in the first image; and Generate first mask data including the union of the first bounding box and the second bounding box, wherein the latent diffuse text to image model masks image data in the first image defined by the first mask data for a filling task.

15. The system of any one of claims 9 to 10, wherein the non-transitory computer-readable storage memory stores instructions, which, when executed by the at least one processor, are effective for: Determine a second bounding box representing the position of the first object in the first image; and Generate one or more second anchor words that include text indicating that the second bounding box is the background.

Images