Video editing using diffusion model
Through fine-tuning of the video diffusion model and text-based video editing technology, the problems of video object movement and appearance modification in the existing technology are solved, and high-quality, time-consistent output video is generated, reducing costs and improving Editing efficiency.
Patent Information
- Application Number
- CN202480001938.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-01-31
- Filing Date
- 2024-01-31
- Publication Date
- 2024-11-05
AI Technical Summary
Existing video processing technologies are difficult to effectively implement custom modifications to the movement and appearance of objects in input videos, especially when maintaining time consistency, and are costly.
Use the video diffusion model for fine-tuning, combined with text-based video editing technology, and generate a downgraded version by downsampling and noise processing of the input video, and update the intermediate representation using multiple reverse diffusion steps to generate high-quality output video to ensure output The video is aligned with the time information of the input video.
High-quality customized modifications to the movement and appearance of objects in the video are realized, ensuring that the output video has temporal consistency on multiple video frames, and improving the quality of video editing and the alignment of content with text prompts.
Smart Images

Figure CN118901100A_ABST
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Application No. 63 / 442,343, filed on January 31, 2023. The disclosure of the prior application is considered part of the disclosure of the present application and is incorporated by reference into the disclosure of the present application. Background Art
[0002] This specification relates to video processing using neural networks.
[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values of the corresponding set of parameters. Summary of the invention
[0004] This specification describes how a conditional video generation system implemented as a computer program on one or more computers at one or more locations generates an output video from a system input. The conditional video generation system is a system that facilitates text-based appearance or motion editing of objects depicted in an input video. A video includes a plurality of video frames, each of which includes a plurality of pixels. Each pixel in each video frame has one or more intensity values.
[0005] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method for generating an output video, the method comprising: obtaining an input video; obtaining input text including a description of the output video; generating a degraded version of the input video based at least on applying downsampling to the input video; and generating the output video by updating the degraded version of the input video based on the description in the input text, wherein at each of a plurality of steps, the updating comprises: processing a diffusion model input including (i) a current intermediate representation of the output video and (ii) the input text by a diffusion model to generate a noisy output for the step; and denoising the current intermediate representation of the output video using the noisy output to generate an updated intermediate representation of the output video of the step.
[0006] In some examples, the conditional video generation system includes a video editing system.
[0007] In some cases, the system input includes an input video and input text. The input video may include a time sequence of video frames showing any of various types of objects, including landmarks, landscape or location features, vehicles, tools, food, clothing, devices, animals, to name a few examples. The input text may include a text prompt that describes the output video, such as describing one or more desired attributes or characteristics that an object displayed in the output video should have.
[0008] For example, the textual prompt may define or otherwise specify that the output video should display additional objects that were not displayed in the input video (or vice versa, i.e., the output video should omit existing objects displayed in the input video). As another example, the textual prompt may define or otherwise specify that an object displayed in the output video should have a different visual appearance than an object displayed in the input video. As another example, the textual prompt may define or otherwise specify that an object displayed in the output video should have a different motion than an object displayed in the input video, i.e., the input video and the output video each display the object with a different continuous motion starting from a start frame to an end frame of the video.
[0009] In these cases, the system processes the system input and generates an output video from the input video under the guidance of the input text. Therefore, the output video generated by the system includes a sequence of video frames that not only reflects the input text but also ensures temporal consistency between the input video and the generated output video frames.
[0010] In other cases, the system input includes an input image or a single video frame, and an input text prompt describing the output video. The input image can similarly display any of the various types of objects described above. In these other cases, the system additionally employs a video synthesis process, in which the system generates a composite video having multiple frames from the input image, for example, by applying repetition, replication, perspective transformation to the input image, instead of or in addition to other image processing operations. The composite video is used as the input video, which is then processed by the system to generate the output video.
[0011] The system may obtain system input in any of a variety of ways. For example, the system may receive input video and / or input text as uploaded content from a remote user of the system over a data communications network, e.g., using an application programming interface (API) provided by the system. As another example, the system may receive input from a user specifying which video data already maintained by the system or another system accessible by the system should be used as input video from which an output video is to be generated.
[0012] A video editing system generates an output video conditioned on the system input using a video diffusion model. Certain implementations of the system may use a video diffusion model that, instead of predicting the output video frame by frame, jointly models the entire video or blocks of frames to improve the temporal coherence between the input video and the generated frames.
[0013] Before using the video diffusion model to generate an output video, the video editing system fine-tunes the model with respect to the input video based on an optimized hybrid fine-tuning objective, i.e., determines updates to pre-trained parameter values of the video diffusion model to improve the quality of motion editing of the input video. Specifically, by keeping the pre-trained parameter values of the temporal attention layer in the model fixed (e.g., by masking) while allowing the pre-trained parameter values of the spatial attention layer in the model to be updated, the video editing system fine-tunes the video diffusion model to reconstruct individual frames of the input video while discarding information about the temporal order of the frames.
[0014] Generating an output video using a video diffusion model typically involves performing a sequence of multiple back-diffusion steps to iteratively update (i.e., denoise) an intermediate (i.e., noisy) representation of the video based on the noise term computed by the model up to that step. However, rather than initializing such an intermediate representation by sampling from a noise distribution (e.g., a Gaussian noise distribution) to determine the intensity value of each pixel in each video frame, the described video editing system initializes the intermediate representation by applying downsampling to the input video and, in some cases, adding noise to the input video to generate a degraded version of the input video. In this way, the first back-diffusion step in the sequence is performed on a degraded version of the input video, which, despite its low resolution, still contains spatiotemporal information from the original input video that facilitates the generation of a higher quality output video.
[0015] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.
[0016] The techniques described in this specification extend the use of a video diffusion model (which is advantageously configured to jointly model entire videos or frame blocks) to text-based video editing, i.e., operable to process an input video and a text prompt to generate an output video that reflects the text prompt from the input video. By fine-tuning the video diffusion model to optimize a hybrid fine-tuning loss function and then configuring the fine-tuned model to generate the output video by performing multiple diffusion steps starting from a degraded version of the input video rather than, for example, simply starting from random noise, the described techniques ensure that high-resolution details such as fine textures or object identities are preserved in the output video, and combine low-resolution spatiotemporal information from the input video with synthesized high-resolution information generated during inference using the model to improve alignment of content in the output video with the text prompt.
[0017] The described techniques enable customized modifications to motion or appearance, in particular both motion and appearance of objects depicted in an input video. Because the described techniques facilitate the generation of smooth visual modifications that align with temporal information in the input video, the output video is a high-quality video that displays objects with desired motion and / or appearance and with temporal consistency over multiple video frames. The described techniques enable new applications that were previously difficult or costly to implement in the field of computer vision, including animation of objects / backgrounds in static images, and creation of dynamic camera motions, to name a few examples.
[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1A is a diagram illustrating an example architecture for fine-tuning a video diffusion model.
[0020] Figure 1B is a diagram illustrating an example architecture for performing inference using a fine-tuned video diffusion model.
[0021] Figure 2 is a flow chart of an example process for generating an output video.
[0022] Figure 3 yes Figure 2 A flowchart of a process step with sub-steps within a step.
[0023] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0024] Figure 1A and Figure 1BAn example architecture for fine-tuning and performing inference using a video diffusion model is shown, respectively. Specifically, Figure 1A 1 is a diagram illustrating an example architecture 100a for fine-tuning a video diffusion model 102. The architecture 100a includes a fine-tuning system 120. The components of the fine-tuning system 120 may be implemented by a computing system including one or more computers that coordinate to fine-tune the video diffusion model 102.
[0025] The video diffusion model 102 may be any suitable diffusion neural network that has been pre-trained, for example by the fine-tuning system 120 or another training system, to generate an output video by performing a back-diffusion process in a plurality of back-diffusion steps.
[0026] In some cases, the video diffusion model 102 may include a sequence (or "cascade") of a low-resolution video diffusion model and a high-resolution video diffusion model, the high-resolution video diffusion model being configured to generate a high-resolution video (where each video frame has a relatively higher resolution) as an output video conditioned on a low-resolution video (where each video frame has a relatively lower resolution) generated by the low-resolution video diffusion model. By utilizing a sequence of video diffusion models, the video diffusion model 102 may iteratively upscale the resolution of the video, thereby ensuring that a high-resolution video can be generated without a single model directly generating the video at the desired output resolution.
[0027] At each back-diffusion step, the video diffusion model 102 is configured to process a diffusion model input including a current intermediate (e.g., noisy) representation of the output video according to pre-trained values of the parameters to generate a noise output, and to update (e.g., denoise) the current intermediate representation using the noise output to generate an updated (e.g., denoised) intermediate representation. For example, the noise output may be an estimate of the noise that needs to be added, for example, to a video generated by the system 100 to generate the current intermediate representation of the video.
[0028] In some cases, the computing system may be a distributed computing system including a plurality of computers. However, in other cases, because the fine-tuning process utilizes a relatively small number of images, i.e., a small number of images compared to the large number of images required during the pre-training process, the computing system may include hardware with much lower computational costs, such as a desktop computer, a laptop computer, or a mobile computing device.
[0029] A video (hereinafter also referred to as a "video clip") includes a plurality of video frames, each of which includes a plurality of pixels. Each pixel in each video frame has one or more intensity values. In some cases, the video diffusion model 102 is configured to generate a video by predicting frame by frame. For example, the video diffusion model 102 can generate a video of indefinite length (i.e., including a varying number of frames) by autoregressively predicting the next frame of the video. In other cases, instead of predicting each individual frame, the video diffusion model 102 is configured to jointly model the entire video or frame block. In these other cases, the temporal coherence between the generated frames, the perceptual quality of the generated frames, or both can be improved.
[0030] For example, the video diffusion model 102 may have been trained on a set of training images based on optimizing a pre-trained objective function defined as:
[0031] In equation (1), Denotes a video diffusion model 102, which has a set of parameters and is configured to receive a diffusion model input, the diffusion model input comprising (i) a true value video The noisy representation (ii) Identify the time step (iii) text prompts and (iv) condition video (For example, the true value video being predicted by the video diffusion model 102 ) and is configured according to a set of parameters Process the diffusion model input to generate a noisy output that can be used to generate ground truth video An updated (e.g., denoised) intermediate representation of . is derived from a noise distribution (e.g., a Gaussian distribution ). Noisy representation of the true value video Can be Given, where and is the time step The noise level at the
[0032] Example architectures for video diffusion models and techniques for training such models are described in more detail in Jonathan Ho, et al., Imagen video: High definition video generationwith diffusion models. arXiv preprint arXiv:2210.02303 (2022), Jonathan Ho, et al., Video diffusion models. arXiv preprint arXiv:2204.03458, 2022, and Lijun Yu, et al., Magvit: Masked generative video transformer. arXiv preprint arXiv:2212.05199. arXiv:2212.05199), and Uriel Singer, et al., Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022), the entire contents of which are hereby incorporated by reference into this article in their entirety.
[0033] The fine-tuning system 120 obtains a fine-tuning dataset 130 and uses the fine-tuning dataset 130 to fine-tune the pre-trained video diffusion model 102, i.e., to update the pre-trained values of the parameters of the video diffusion model 102. The fine-tuning dataset 130 includes a plurality of video clips 132, wherein each video clip 132 includes a plurality of consecutive video frames. The fine-tuning dataset 130 also includes a plurality of unordered video frames 134. Each video frame 134 may be an individual image.
[0034] In some cases, multiple video clips 132 each depict a specific subject instance of an object class (rather than different subject instances of the object class). Likewise, in some cases, multiple unordered video frames 134 each depict a specific subject instance of an object class (rather than different subject instances of the object class). Generally speaking, there may be multiple subject instances belonging to a common object class. For any object class, each subject instance belonging to the object class may have a set of appearance characteristics that visually distinguish the subject instance from other subject instances belonging to the same object class. In other words, although different subject instances all belong to the same object class, they may appear different from each other.
[0035] The fine-tuning system 120 may obtain the fine-tuning dataset 130 in any of a variety of ways. For example, the system may receive a plurality of video clips 132 as uploaded content from a remote user of the system via a data communication network, for example using an application programming interface (API) provided by the system, and randomly shuffle a plurality of consecutive video frames included in each of the one or more video clips to generate a plurality of unordered video frames 134. As another example, the system may receive input from a user specifying which video data already maintained by the system or another system accessible by the system should be used as the plurality of video clips 132, and randomly shuffle a plurality of consecutive video frames included in each of the one or more video clips to generate a plurality of unordered video frames 134.
[0036] Specifically, the fine-tuning system 120 fine-tunes the video diffusion model 102 in multiple fine-tuning steps by optimizing a hybrid fine-tuning objective based on video segments 132 and individual video frames 134 sampled from the fine-tuning dataset 130. The hybrid fine-tuning objective function includes a video segment reconstruction loss term, which is a loss term for each video segment sampled from the plurality of video segments 132 included in the fine-tuning dataset 130. Evaluate (i) the video clip and (ii) the reconstructed representation of the video segment generated by using the video diffusion model 102. In other words, the video segment reconstruction loss term trains the video diffusion model 102 to reconstruct the entire video segment including a plurality of consecutive video frames.
[0037] For example, the video segment reconstruction loss term can be defined as follows:
[0038] Equation (2) differs from equation (1) above at least in that, in some cases, during the fine-tuning process, the textual cues included in the diffusion model input to the video diffusion model 102 include a unique identifier. That is, indicates a textual hint when a unique identifier is included, and represents the text hint when such unique identifier is not included. In some other cases where the text hint does not include a unique identifier, in equation (2) becomes .
[0039] The unique identifier identifies a particular instance of a subject depicted in the video clip. The unique identifier may be represented as a string in a given text encoding format, such as Unicode format, ASCII format, or another text encoding format.
[0040] In general, the unique identifiers of different subject instances will be different. That is, a first unique identifier of a first subject instance may include a different token than a second unique identifier of a second subject instance. For example, a first subject instance may be a vehicle, and a second subject instance may be an animal. As another example, a first subject instance may be a vehicle having a first shape / size / color, and a second subject instance may be a vehicle having a second shape / size / color.
[0041] To generate such a unique identifier, the fine-tuning system 120 can, for example, use the techniques described in Nataniel Ruiz, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023, the contents of which are incorporated by reference in their entirety into this specification.
[0042] The hybrid fine-tuning objective function also includes an individual frame reconstruction loss term, which is calculated for each individual video frame sampled from the plurality of unordered video frames 134 included in the fine-tuning dataset 130. Evaluate (i) the video frame and (ii) the reconstructed representation of the video frame generated by using the video diffusion model 102. In other words, the individual frame reconstruction loss term trains the video diffusion model 102 to reconstruct individual video frames (rather than entire video clips).
[0043] For example, the individual frame reconstruction loss term can be defined as follows:
[0044] In equation (3), represents an individual video frame sampled from the plurality of unordered video frames 134 included in the fine-tuning dataset 130. For example, the individual video frame It can be a plurality of video clips included in the fine-tuning dataset 130 Any frame in a video clip. represents the video diffusion model 102, which in some cases has a modified architecture in which its temporal attention layer is masked (as explained below). In some other cases where the architecture is not modified, in equation (3) become .
[0045] For example, at each fine-tuning step, the fine-tuning system 120 may update the parameters of the video diffusion model 102 based on optimizing a hybrid fine-tuning objective including both a video segment reconstruction loss term and an individual frame reconstruction loss term. to determine the parameter according to the following The updated (eg, fine-tuned) value of :
[0046] In equation (4), It is the weighted combination of the video segment reconstruction loss and the individual frame reconstruction loss. May correspond to an adjustable hyperparameter of fine-tuning system 120 .
[0047] During fine-tuning, the fine-tuning system 120 may incorporate any number of techniques to improve the effectiveness, efficiency, or both of the fine-tuning process. For example, the fine-tuning system 120 may use a small number of fine-tuning steps ( ), low learning rate (For example, approximately or lower) or both, combined with a specific choice of weight values to reduce overfitting. Some exemplary combinations of fine-tuning steps and weight values are given below:
[0048] Furthermore, depending on the configuration of the video diffusion model 102, such as whether it has a Transformer-based architecture or a convolutional architecture, or whether it is generated autoregressively one video frame after another or jointly outputs the entire video, the fine-tuning system 120 may combine different techniques when fine-tuning the video diffusion model 102.
[0049] In some cases, the architecture of the pre-trained video diffusion model 102 remains unchanged during fine-tuning. However, in other cases, the architecture of the pre-trained video diffusion model 102 is modified during fine-tuning, for example, by adding one or more additional layers to replace or in addition to existing layers of the pre-trained video diffusion model 102.
[0050] In some cases, all parameters of the video diffusion model 102 are adjusted during fine-tuning. In other cases, only some parameters of the video diffusion model 102 are updated, while other parameters of the video diffusion model 102 are kept fixed to their pre-trained values. As a specific example, parameters of some layers of the video diffusion model 102 are kept fixed, and only parameters of some other layers of the video diffusion model 102 are updated.
[0051] As a specific example, the video diffusion model 102 can have a Transformer-based architecture that includes one or more temporal attention layers, one or more spatial attention layers, and one or more convolutional layers. The temporal attention layer is a layer that includes an attention mechanism (e.g., a query-key-value (QKV) attention mechanism) and focuses on multiple video frames in a video when generating a corresponding temporal attention layer output from a temporal attention layer input. The spatial attention layer is a layer that includes an attention mechanism and focuses on multiple pixels in a video frame when generating a corresponding spatial attention layer output from a spatial attention layer input. The convolutional layer is a layer that applies a convolutional filter across multiple pixels in a video frame when generating a corresponding convolutional layer output from a convolutional layer input.
[0052] In this example, the fine-tuning system 120 can adjust the pre-trained parameter values of one or more spatial attention layers while keeping the pre-trained parameter values of one or more temporal attention layers and one or more convolutional layers fixed when fine-tuning the video diffusion model 102 based on optimizing the individual frame reconstruction loss term. For example, this can be achieved by inserting a mask before each temporal attention layer and each convolutional layer included in the video diffusion model 102.
[0053] therefore, Figure 1A As shown, when based on the video segments sampled from the plurality of video segments 132 included in the fine-tuning dataset 130 by processing When the video diffusion model 102 is fine-tuned to optimize the video segment reconstruction loss term, the video diffusion model 102 has its original, unmodified architecture (where the temporal attention layer and the convolutional layer are not masked). Alternatively, when the video diffusion model 102 is fine-tuned based on the individual video frames sampled from the plurality of unordered video frames 134 included in the fine-tuning dataset 130, the video diffusion model 102 has its original, unmodified architecture (where the temporal attention layer and the convolutional layer are not masked). When fine-tuning the video diffusion model 102 to optimize the individual frame reconstruction loss terms, the video diffusion model 102 has a modified architecture (in which the temporal attention layers and convolutional layers are masked).
[0054] Specifically, by keeping the pre-trained parameter values of the temporal attention layers and convolutional layers in the video diffusion model 102 fixed (e.g., by masking) while allowing the pre-trained parameter values of the spatial attention layers in the video diffusion model 102 to be updated, the fine-tuning system 120 fine-tunes the video diffusion model 102 to reconstruct individual frames sampled from the fine-tuning dataset 130 while discarding information about the temporal order of these frames.
[0055] Once fine-tuning is complete, e.g., after a predetermined number of fine-tuning steps have been performed, the fine-tuning system 120 can provide data specifying the fine-tuned video diffusion model 102, i.e., data specifying fine-tuned parameter values, and in some cases data specifying the architecture of the video diffusion model 102, for deployment on another system to perform inference, e.g., for conditional video generation or video content editing. Alternatively or in addition, the fine-tuning system 120 can deploy the fine-tuned video diffusion model 102 and use the video diffusion model to generate new videos in response to user requests.
[0056] Figure 1B is a diagram illustrating an example architecture 100b for performing inference using a fine-tuned video diffusion model 122. The architecture 100b includes a conditional video generation system 140 implemented by a computing system including one or more computers, the conditional video generation system including the fine-tuned video diffusion model 122, parameters of which have been adjusted according to a fine-tuning process.
[0057] The conditional video generation system 140 may use the fine-tuned video diffusion model 122 to generate an output video 160 conditioned on system input 150 provided by a user of the system (eg, via a client device).
[0058] In some cases, system input 150 includes input text 151. Input text 151 may include, for example, a text prompt in natural language that describes output video 160, such as describing one or more desired attributes or characteristics that an object displayed in the output video should have.
[0059] In some of these cases, the text hint may include a unique identifier that identifies a particular subject instance, such as one of the subject instances depicted in the plurality of video clips 132 and / or the plurality of unordered video frames 134 included in the fine-tuning dataset 130 used in the fine-tuning process. As described above, the unique identifier may be represented as a string in a given text encoding format, such as a Unicode format, an ASCII format, or another text encoding format.
[0060] In some cases, system input 150 includes input video 152. Input video 152 may include a time sequence of video frames showing any of various types of objects, including landmarks, landscape or location features, vehicles, tools, food, clothing, devices, animals, to name a few examples.
[0061] In some cases, system input 150 includes input image 153 or a single video frame. Input image 153 may similarly display any of the various types of objects described above.
[0062] In some cases, system input 150 includes two or more of the above data items. For example, system input 150 includes both input text 151 and input video 152, wherein input text 151 specifies the desired edits or modifications that need to be made to input video 152. Specifically, input text 151 may include a text prompt that defines or otherwise specifies that output video 160 should display additional objects that are not displayed in input video 152 (or vice versa, i.e., output video 160 should omit existing objects displayed in input video 152). Additionally or alternatively, input text 151 may include a text prompt that defines or otherwise specifies that output video 160 should display new objects to replace existing objects displayed in input video 152. Additionally or alternatively, input text 151 may include a text prompt that defines or otherwise specifies that objects displayed in output video 160 should have a visual appearance different from the visual appearance of objects displayed in input video 152. Additionally or alternatively, input text 151 may include a text prompt that defines or otherwise specifies that an object displayed in output video 160 should have a motion that is different from the motion of an object displayed in input video 152, i.e., the input video and the output video each display the object having a different continuous motion from the start frame to the end frame of the video.
[0063] As another example, system input 150 includes both input text 151 and input image 153, where input text 151 specifies how output video 160 should be generated based on input image 153 (e.g., based on one or more objects depicted in input image 153). For example, input text 151 may specify that output video 160 should depict the same subject instance depicted in input image 153, e.g., with a particular motion, etc., in addition to other background objects.
[0064] After obtaining the system input 150, the conditional video generation system 150 can then process the system input 150 and generate an output video 160 by performing a back-diffusion process across multiple back-diffusion steps conditioned on the system input 150 using the fine-tuned video diffusion model 122. Figures 2 to 3 Generating the output video 160 using the fine-tuned video diffusion model 122 is described in more detail. The conditional video generation system 150 can then provide the output video 160 for presentation to a user who provided the system input 150 (eg, on a client device).
[0065] Output video 160 includes a sequence of video frames. Depending on what is included in system 150, the video frames included in output video 160 may depict any of a variety of content.
[0066] For example, when the system input 150 includes both the input video 152 and the input text 151 specifying the desired edits or modifications that need to be made to the input video 152, the output video 160 can be an edited or modified version of the input video 152 having the desired edits or modifications, e.g., displaying additional objects, omitting existing objects, replacing existing objects with new objects, displaying objects with different visual appearances, displaying objects with different motions, etc. As a specific example, both the output video 160 and the input video 152 depict the same subject instance, however, the motion, visual appearance, or both of the subject instance are different.
[0067] Specifically, in this example, through the fine-tuning process, the output video 160 includes a sequence of video frames that not only reflects the desired edits or modifications specified by the input text 151, but also ensures temporal consistency between the frames included in the input video 152 and the frames included in the output video 160.
[0068] As another example, when the system input 150 includes both the input image 153 and the input text 151 specifying how the output video 160 should be generated based on the input image 153, the output video 160 may reflect both the input text 151 and the input image 153. For example, the output video 160 may depict the same subject instance depicted in the input image 153, e.g., with a particular motion specified by the input text 151, etc., in addition to other background objects specified by the input text 151.
[0069] As yet another example, when system input 150 includes input text 151, which in turn includes a unique identifier that identifies a specific subject instance of an object class, output video 160 may depict, for example, the specific subject instance (rather than a different subject instance of the object class) having a specific motion, etc., also specified by input text 151, in addition to other background objects also specified by input text 151.
[0070] Figure 2 is a flow chart of an example process 200 for generating an output video. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed conditional video generation system, such as Figure 1B The conditional video generation system 150 can perform process 200.
[0071] The system obtains an input video comprising a plurality of video frames (step 202). In some cases, the system may receive the input video from a client device, for example as an uploaded content. In these cases, the input video may include a temporal sequence of video frames, i.e., where the plurality of video frames are arranged in a temporal order. In other cases, the system may receive an input image from a client device and then perform a video synthesis process to generate a composite video having a plurality of frames from the input image. The composite video will then be used as the input video.
[0072] Specifically, performing the video synthesis process can involve applying any of a variety of conventional image processing operations to the input images to generate corresponding output images to include as frames of the synthesized video. For example, the system can apply a replication operation to the input images to generate one or more replicated input images, and include the one or more replicated input images as frames of the synthesized video. As another example, the system can apply a perspective transformation to the input images to generate one or more transformed input images, and include the one or more transformed input images as frames of the synthesized video. Other image processing operations can also be used.
[0073] The system obtains input text (step 204). For example, the input text may be provided by the same client device that also provides the input video (or provided by the same client device that also provides the input video from which the input video is generated). The input text may include a description of the output video, such as describing one or more desired attributes or characteristics that an object displayed in the output video should have. Additionally or alternatively, the input text may specify desired edits or modifications that need to be made to the input video.
[0074] The system initializes the output video, that is, generates an initial intermediate representation of the output video based on the input video (step 206). Specifically, the system achieves this by applying downsampling to the input video to generate a downsampled version of the input video, wherein the frames included in the downsampled input video will have a lower resolution than the frames included in the input video; and then adding noise (e.g., Gaussian noise with a predetermined variance) to the downsampled input video to generate a degraded version of the input video. The degraded version of the input video is then used as the initial intermediate representation of the output video.
[0075] By doing so, the system ensures that the first back-diffusion step in the back-diffusion process is performed on a downgraded version of the input video which, despite its low resolution, still contains spatiotemporal information from the original input video that facilitates the generation of a higher quality output video.
[0076] This is in contrast to some conventional backdiffusion processes where such an initial intermediate representation is generated from pure noise, for example by sampling from a noise distribution (eg, a Gaussian noise distribution) to determine the intensity value of each pixel in each frame included in the output video.
[0077] The system generates an output video by updating the initial intermediate representation of the output video based on the description in the input text (step 208). The output video generated by the system will include a temporal sequence of video frames, i.e., a plurality of video frames arranged in a temporal order. For example, the output video may be an edited or modified version of the input video with a desired edit or modification, such as displaying additional objects, omitting existing objects, replacing existing objects with new objects, displaying objects with different visual appearances, displaying objects with different motions, and the like.
[0078] Reference below Figure 3 Generating the output video is described in more detail, Figure 3 Sub-steps 302-304 of step 208 are shown. The system can generate an output video by performing iterations of sub-steps 302-304 at each step in the plurality of back-diffusion steps. In other words, the final output video is generated after the last back-diffusion step in the plurality of back-diffusion steps.
[0079] The system processes diffusion model inputs including (i) a current intermediate representation of the output video, (ii) input text, (iii) the input video, and (iv) data identifying a time step (corresponding to the current back-diffusion step) through the video diffusion model to generate a noise output for that step (step 302). For example, the noise output can be an estimate of the noise that needs to be added to the output video to generate the current intermediate representation of the output video and can be used to generate a prediction of the output video given the current intermediate representation. It should be understood that processing the current intermediate representation of the output video includes processing pixels of one or more frames of the intermediate representation of the output video.
[0080] For example, the time steps may run backwards from 1 to 0. For the first back-diffusion step, the current intermediate representation is the initial intermediate representation of the output video that has been generated in step 206. For each subsequent back-diffusion step, the current intermediate representation is the updated intermediate representation generated in the previous back-diffusion step.
[0081] The system uses the noise output to denoise the current intermediate representation of the output video to generate an updated intermediate representation of the output video of this step (step 304). The system can update the current intermediate representation by applying the diffusion sampler to the current intermediate representation. Applying the diffusion sampler to the current intermediate representation produces an updated intermediate representation that has the same dimensions as the current intermediate representation, but has different values, i.e., updated values.
[0082] For example, applying a diffusion sampler can include using a DDIM sampler with random noise correction. At each step, the expected denoised image can be calculated and used to estimate the noise. For example, a portion of the estimated noise can be removed, and randomly generated Gaussian noise can be added, the magnitude of which corresponds to half of the removed noise. The DDIM sampler is described in more detail in Jonathan Ho, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. Other suitable samplers, such as ancestral samplers, may also be used.
[0083] This specification uses the term "configuration" in connection with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform that operation or action. For one or more computer programs to be configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.
[0084] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), generating an artificially generated propagation signal to encode information for transmission to a suitable receiver device for execution by a data processing device.
[0085] The term "data processing equipment" refers to data processing hardware and encompasses all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or include a dedicated logic circuit system, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0086] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0087] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed in a different way.
[0088] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0089] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.
[0090] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a dedicated logic circuit system or incorporated into the dedicated logic circuit system. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from a large-capacity storage device or transfer data to a large-capacity storage device or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0091] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.
[0092] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving responsive messages from the user in response.
[0093] A data processing device used to implement a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing general-purpose and computationally intensive portions of machine learning training or production (i.e., inference, workloads).
[0094] Machine learning models can be implemented and deployed using a machine learning framework (e.g., TensorFlow or JAX framework).
[0095] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0096] A computing system may include a client and a server. The client and server are usually remote from each other and typically interact through a communication network. The relationship of client and server is due to computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for displaying data to a user interacting with a device acting as a client and receiving user input from it. Data generated at a user device, for example, due to user interaction, may be received at the server from the device.
[0097] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.
[0098] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0099] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or ordered order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method for generating an output video, the method comprising: Get input video; obtaining input text including a description of an output video; generating a degraded version of the input video based at least on applying downsampling to the input video; as well as The output video is generated by updating the degraded version of the input video based on the description in the input text, wherein at each of a plurality of steps, the updating comprises: processing, by a diffusion model, a diffusion model input comprising (i) a current intermediate representation of the output video and (ii) the input text to generate a noisy output of the step; as well as The current intermediate representation of the output video is denoised using the noise output to generate an updated intermediate representation of the output video of the step.
2. The method of claim 1 , wherein generating the degraded version of the input video comprises: generating a downsampled version of the input video by applying downsampling to the input video; as well as The degraded version of the input video is generated by adding Gaussian noise having a predetermined variance to the downsampled version of the input video.
3. The method according to any one of claims 1 to 2, wherein obtaining the input video comprises: The input video having a plurality of video frames is received from a client device.
4. The method according to any one of claims 1 to 2, wherein obtaining the input video comprises: receiving one or more input images from a client device; generating a composite video having a plurality of video frames by replicating, transforming, or both, each of the one or more input images; as well as The synthesized video is used as the input video.
5. The method of any one of claims 1 to 4, wherein the diffusion model has pre-trained parameter values, and generating the output video based on the description in the input text comprises: The diffusion model is fine-tuned with respect to the input video to adjust the pre-trained parameter values of the diffusion model.
6. The method of claim 5, wherein fine-tuning the diffusion model with respect to the input video comprises: The pre-trained parameter values of the diffusion model are adjusted based on optimizing a hybrid fine-tuning objective function, the hybrid fine-tuning objective function comprising a first term and a second term, wherein the first term evaluates the difference between (i) the input video and (ii) a reconstructed representation of the input video generated by using the diffusion model, and the second term evaluates the difference between (i) each of one or more frames of the input video and (ii) a reconstructed representation of each of the one or more frames of the input video generated by using the diffusion model.
7. The method of any one of claims 5 to 6, wherein fine-tuning the diffusion model with respect to the input video comprises: generating a unique identifier for an instance of a subject depicted in the input video; as well as The unique identifier is processed as the input text by the diffusion model during the fine-tuning.
8. The method of any one of claims 5 to 7, wherein the diffusion model comprises: (i) one or more temporal attention layers, each of the one or more temporal attention layers focusing on the plurality of video frames in the input video when generating a corresponding attention layer output, and (ii) one or more spatial attention layers, each of the one or more spatial attention layers focusing on a plurality of pixels in a video frame when generating a corresponding spatial attention layer output, and wherein fine-tuning the diffusion model on the input video comprises: The pre-trained parameter values of the one or more spatial attention layers are adjusted while the pre-trained parameter values of the one or more temporal attention layers are kept fixed.
9. The method of any one of claims 1 to 8, wherein the output video and the input video both depict a subject instance, but the motion, appearance, or both of the subject instance are different.
10. A system comprising: One or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding method as claimed in any one of claims 1 to 9.
11. A computer storage medium encoded with a computer program, the program comprising instructions operable when executed by a data processing apparatus to cause the data processing apparatus to perform the operations of a corresponding method as claimed in any one of claims 1 to 9.
Citation Information
Cited By
Video processing method and device
CN121099081A
