Generating videos of agents communicating from still images using generative neural networks

Abstract

Methods and systems for generating videos of agents communicating from still images. For example, an initial image of an agent and a conditioning input characterizing sound emitted by the agent can be obtained and a video spanning a first time period of the agent emitting the sound can be generated from the initial image and the conditioning input using a motion generation neural network and a video generation neural network.

Description

[0001] GENERATING VIDEOS OF AGENTS COMMUNICATING FROM STILL IMAGES USING GENERATIVE NEURAL NETWORKS

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims the benefit of Greece Application No. 20240100047, filed January 25, 2024. The disclosure of the foregoing application is hereby incorporated by reference in its entirety.

[0004] BACKGROUND

[0005] This specification relates processing data using machine learning models.

[0006] As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates a video of an agent, e.g., a human, a robot, an animal, a digital agent, e.g., a character or avatar rendered or displayed in a software application, or other agent, from an initial image of the agent. In other words, the system generates the intensity’ values of the pixels of the video frames at each of multiple time steps within the time window spanned by the video.

[0009] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0010] This specification describes techniques for generating videos of an agent emitting target sounds, e.g., speaking, from a single image of the agent. Making use of the described techniques enables the generation of high-quality videos of variable length, which are easily controllable through high-level representations of agent faces and bodies. In contrast to previous work, the described techniques do not require training for each agent, do not rely on face detection and cropping, and generate complete images (not just the face or the lips) of the agent. Moreover, the descnbed techniques can be used to generate videos that cover a broad spectrum of scenarios (e.g., visible torso or diverse subject identities) that are critical to correctly synthesize, e.g., videos of humans or other agents that are communicating.

[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0012] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0013] BRIEF DESCRIPTION OF DRAWINGS

[0014] FIG. 1 shows an example video generation system.

[0015] FIG. 2 is a flow diagram of an example process for generating a video of an agent emitting sounds.

[0016] FIG. 3 is a flow diagram of an example process for generating a sequence of control images.

[0017] FIG. 4 shows an example of the operation of the system.

[0018] FIG. 5 shows examples of videos generated by the system.

[0019] Like reference numbers and designations in the various drawings indicate like elements.

[0020] DETAILED DESCRIPTION

[0021] FIG. 1 shows an example video generation system 100. The video generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0022] The system 100 is a system that generates a video 112 of an agent, e.g., a human, a robot, an animal, a digital agent, e g , a character or avatar rendered or displayed in a software application, or other agent, from an initial image 102 of the agent. In other words, the system generates the intensity values of the pixels of the video frames at each of multiple time steps within the time window spanned by the video 112.

[0023] In particular, the system 100 receives the initial image 102 of the agent and a sound conditioning input 104, i.e., a conditioning input that characterizes sound emitted by the agent 104 and generates, from the initial image 102 and the sound conditioning input 104, the video 112 of the agent emitting the sound. For example, the system 100 can receive an audio signal of the sound or text characterizing the sound.

[0024] As a particular example, the system 100 can receive a conditioning input 104 characterizing speech of the agent and generate a video of the agent speaking the speech, e.g., a video of a human depicted in the initial image speaking the speech characterized by the conditioning input.

[0025] To generate the video, the system 100 processes the conditioning input 104 using a motion generation neural network 120 to generate motion controls 122 for the agent.

[0026] The motion controls 122 generally include parameters that characterize motion of the agent during the time period spanned by the video, i.e., characterizing motion that would be required for the agent to emit the sound characterized by the conditioning input.

[0027] Specific examples of parameters will be described below.

[0028] The system 100 generates, from the initial image 102 of the agent and the motion controls 122, a sequence of control images 130 of the agent that characterize motion of the agent during the first time period.

[0029] For example, the system 100 can apply a statistical model to the initial image 102 and the motion controls 122 to generate motion of the agent and then generate the control images 130 that capture the generated motion.

[0030] For example, the system 100 can apply the statistical model to the initial image 102 to generate an initial configuration of the agent and then apply the motion controls 120 to the initial configuration to generate the control images.

[0031] Generating the control images will be described in more detail below with reference to FIG. 3.

[0032] The system 100 then processes the initial image 102 and the sequence of control images 130 using a video generation neural network 140 to generate the video 112 of the agent emitting the sound.

[0033] Once generated, the system 100 can use the video 112 for any of a variety of purposes.

[0034] For example, the system 100 can store the video 112 in association with the audio signal for later play back.

[0035] As another example, the system 100 can provide the video 112 for play back along with the audio signal, so that the agent emits the audio while the video 112 is playing.

[0036] The video generation neural network 140 can generally be any appropriate neural network that can map an initial image 102 in the video and a set of conditioning images, i.e., the control images 130, to a video. As a specific example, the video generation neural network 140 can be an image or video diffusion neural network. Specific examples of the architecture of the video generation neural network 140 will be described in more detail below with reference to FIG. 4.

[0037] FIG. 2 is a flow diagram of an example process 200 for generating a video of an agent emitting a sound. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g., the video generation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0038] The system obtains an initial image of an agent (step 202).

[0039] For example, the initial image can be a real-world image of the agent captured by a camera device.

[0040] As another example, the initial image can be a synthetic image of the agent generated using an image generation neural network or another image generation technique.

[0041] The system obtains a conditioning input characterizing sound emitted by the agent (step 204).

[0042] For example, the conditioning input can include an audio signal of the agent emitting the sound.

[0043] As another example, the conditioning input can include text describing the sound. For example, when the sound is speech, the conditioning input can be a text transcription of the speech.

[0044] The system then generates, from the initial image and the conditioning input, a video spanning a first time period of the agent emitting the sound.

[0045] As part of this, the system processes the conditioning input using a motion generation neural network to generate motion controls for the agent (step 206). As described above, the motion controls generally include parameters that characterize motion of the agent during the time period spanned by the video, i.e., characterizing motion that would be required for the agent to emit the sound characterized by the conditioning input.

[0046] For example, the motion controls for the agent can include, for each of a plurality of time points during the first time period, a first set of controls defining facial expression parameters of the face of the agent at the time point. For example, these parameters can identify the positions of various landmarks, e.g.. eyes, nose, mouth, and so on, on the face of the agent as of the time point. That is, the first set of controls defines the facial expression parameters of the face of the agent at the time point.

[0047] As another example, the motion controls for the agent can include, for each of the plurality of time points during the first time period, a second set of controls defining head pose parameters of a head of the agent at the time point. For example, these parameters can define respective positions of a set of predetermined, template vertices of the head of the agent at the time point. That is, the second set of controls defines the pose of the head of the agent at the time point. For example, these can be in the form of displacements, i.e., differences from a set of reference head pose parameters of the head of the agent at the time point.

[0048] As another example, the motion controls for the agent can include, for each of a plurality of time points during the first time period, a third set of controls defining body pose parameters of a body of the agent at the time point. For example, these parameters can define respective positions of a set of predetermined, template vertices of the body of the agent at the time point. That is. the second set of controls defines the pose of the body of the agent at the time point.

[0049] For example, these can also be in the form of displacements, i.e., differences from a set of reference body pose parameters of the body of the agent.

[0050] The motion generation neural network can generally have any appropriate architecture that allows the system to use the neural network to map the conditioning input to the set of motion controls. For example, the motion generation neural network can be a convolutional neural network, e.g., a U-Net. As another example, the motion generation neural network can be a self-attention neural network. As yet another example, the motion generation neural network can include both convolutional and self-attention layers.

[0051] In some implementations, the motion generation neural network is implemented in a diffusion framework, so that the system uses the motion generation neural network to gradually denoise an initial, noisy set of motion controls across multiple denoising steps. That is, in this framework, the system initializes a representation of the motion controls that has the same dimensionality as the motion controls, e.g.. by sampling each value in the representation from a noise distribution, e.g., a Gaussian distribution.

[0052] At each of multiple denoising steps, the system processes the representation of the motion controls and the conditioning input using the motion generation neural network to generate a prediction of the "clean” or "denoised” set of motion of controls. The system then uses the prediction to update the representation, e.g., by applying a diffusion sampler, e.g.. DDPM, DDIM, or another appropriate sampler, to the prediction and the representation.

[0053] After the final denoising step, the system can use the prediction of the clean or denoised set of motion controls as the final set of motion controls.

[0054] As described above, the system can receive different types of conditioning inputs, e.g., audio signals or text describing the audio. In some cases, the system can map each conditioning input to a particular type of representation that is then processed by the motion generation neural network.

[0055] As a particular example, the system can map each type of conditioning input into a spectrogram, e.g., a mel-spectrogram, and then condition the motion generation neural network on the mel-spectrogram. As a particular example of this, the system can receive a representation of an audio signal as a mel-spectrogram. As another particular example of this, the system can receive a representation of an audio signal as a waveform and then convert the waveform into a mel-spectrogram. As another example, the system can receive text and generate an audio signal representing the text using an appropriate machine learning model. The system can then convert the audio signal representing the text into a mel-spectrogram. For example, when the text is a text transcription of speech spoken by the agent, the system can use a text-to-speech (TTS) machine learning model to generate the audio signal.

[0056] The system or another training system can train the motion generation neural network in any of a variety of ways.

[0057] For example, the system can obtain a dataset of videos of objects emitting sounds. The system can then extract, from each video in the dataset, target motion controls, e.g., by using a parametric model, a neural network trained through unsupervised learning, or both to extract facial expression parameters and poses from the frames in the video. The system also maps the sounds emitted during the video to respective sound conditioning inputs, e.g., as described above. The system can then train the motion generation neural network input to map the target conditioning inputs to the corresponding target motion controls. For example, when the motion generation neural network is used as part of a diffusion framework, the system can train the motion generation neural network on an appropriate diffusion model training objective, e.g., a score matching loss.

[0058] For example, a motion generation neural network can be trained using a diffusion framework which progressively adds Gaussian noise e ~ N (0, 1) to ground-truth samplesxo with a conditional audio input, a. The goal is to model the motion distribution of real heads and bodies, xo ~ q(xo\a), by training a denoising network that predicts the added noise from the noisy input xt, where t is an arbitrary diffusion step. The goal of the loss function is, in some examples, to directly predict the ground truth distribution, e.g.:

[0059] In some examples a further temporal loss is also included to penalize prediction difference at consecutive frames, for any given frame i G N. The model can be trained using a linear combination of both losses, e g., Different temporal loss weights, Atemp, can be used for expressions and body poses to ensure smoother motion for the head and hands while allowing larger dynamism for facial expressions.

[0060] The system generates, from the initial image of the agent and the motion controls, a sequence of control images of the agent that characterize motion of the agent during the first time period (step 208).

[0061] As a particular example, the control images can include, at each of the plurality of time points during the first time period, a control image characterizing the pose of the agent at the time point. For example, each control image can be a respective mask image that identifies positions of particular portions, e.g., joints or other locations, on the body, head, or both of the agent.

[0062] As another example, the control images can include, at each of the plurality of time points during the first time period, a control image characterizing positions of semantic regions of the head of the agent at the time point. For example, each control image can be a respective mask image that identifies the semantic regions of the head of the agent at the timepoint.

[0063] For example, the system can use a statistical model to generate the control images.

[0064] An example of this is described below with reference to FIG. 3.

[0065] The system processes the initial image and the sequence of control images using a video generation neural network to generate the video of the agent emitting the sound (step 210).

[0066] For example, the video generation neural network can be a diffusion neural network. As a particular example of this, the video generation neural network can be a temporal image diffusion neural network or can include a temporal image diffusion neural network. An example of the video generation neural network being a temporal image diffusion neural network is described above.

[0067] FIG. 3 is a flow diagram of an example process 300 for generating a sequence of control images. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g.. the video generation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0068] The system applies, to the initial image of the agent, a statistical model to generate parameters of an initial configuration of the agent in the initial image (step 302). Applying a statistical model can refer to fitting a statistical model on to the initial image. For example, the statistical model can be the SPHEAR model, as described in Eduard Gabriel Bazavan, Andrei Zanfir, Teodor Alexandru Szente, Mihai Zanfir, Thiemo Alldieck, and Cristian Sminchisescu. Sphear: Spherical head registration for complete statistical 3d modeling. 3DV, 2024.

[0069] In this example, the parameters generated by the model include reference head pose parameters and a shape code / ?sthat defines the reference shape of the head of the agent.

[0070] Optionally, the system can also use a pose estimation neural network, e.g., one trained through unsupervised learning, to generate reference body pose parameters from the initial image.

[0071] The system can then apply the motion controls to the initial configuration to generate updated parameters at each of a plurality of time points during the first time point (step 304). For example, for each of the plurality of time points, the system can set the updated parameters that correspond to the facial expression equal to the corresponding parameters in the motion controls. For the parameters that correspond to head and body pose, the system can add or subtract the corresponding parameters in the motion controls from the corresponding reference parameters.

[0072] For each time point, the system can then generate one or more control images from the updated parameters at the time point (step 306).

[0073] For example, the system can rasterize the template vertex positions of the posed heads as dense representations, and the body as sparse joint representations, to obtain a respective dense mask for each time point that serves as one of the control images for the time point. As another example, the system can rasterize the semantic regions of the posed heads to generate another dense mask for each time point that serves as another one of the control images for the time point.

[0074] FIG. 4 shows an example 400 of the operation of the system.

[0075] As shown in the example 400, the system receives an input image Iref and an audio input a. The system processes the audio input a using the motion generation neural network A to generate motion controls.

[0076] The system also fits a statistical model to the input image Irefto generate shape parameters and the reference pose. The system then uses the motion controls and the shape parameters and the reference pose to generate the control images. In the example 400, the control images include a set of dense facial expression image controls C (e.g., G j^wxi xs^anc|a set opsemanpc masks C™ for relevant facial regions (e.g., e{0,l}"x™ forNCdifferent semantic classes).

[0077] The system then processes the input image and the control images using the video generation neural network to generate the video, i.e.. that includes a set of generated images Gj that span the future time window.

[0078] In the example 400, the video generation neural network includes a temporal image diffusion neural netw ork and a sequence of one or more super resolution neural netw orks.

[0079] As shown in the example 400, the temporal image diffusion neural network includes a first branch of layer blocks that, for each of a plurality of time points in the video, is configured to receive a first input comprising a current representation of a video frame at the time point in the video and to process the first input to generate a denoising output for updating the current representation.

[0080] The temporal image diffusion neural network also includes a second branch of layer blocks that, for each of the plurality of time point in the video, is configured to receive a second input that includes the respective control images corresponding to the time point and to process the second input to generate a respective layer output for each layer block in the second branch. In some cases, the second input can also include the current representation of the control images.

[0081] Generally, one or more layer blocks in the second branch include a temporal layer that, for each time point, receives inputs from (i) a corresponding layer block in the first branch, (ii) a corresponding layer block in the second branch, or (iii) or both for one or more other time points. That is, the temporal layer incorporates information from other time points to ensure consistency between the video frames in the video. For example, as shown in the example 400, the temporal layer can be a temporal convolutional layer. More specifically, in the example 400, the layer blocks in the second branch are convolutional blocks that include multiple convolutional layers, one of which is a temporal convolutional layer. As a particular example, each layer block can include a group normalization layer, followed by a (spatial) convolutional layer, followed by an activation function layer, e.g.. a Swish layer, followed by the temporal convolutional layer, followed by a group normalization layer, followed by a (spatial) convolutional layer, and followed by an activation function layer, e.g., a Swish layer. The layer blocks in the first branch can, in some cases, have a similar architecture, but without any temporal layers, i.e., without any temporal convolutional layers.

[0082] To condition the generation of the denoising output on the output of the temporal layer, one or more layer blocks in the first branch each receive input from a corresponding layer block in the second branch.

[0083] To generate a set of video frames using the video generation neural network when the neural network includes the temporal diffusion neural network, the system initializes a representation of the set of video frames that has the same dimensionality as the set of video frames, e.g., by sampling each value in the representation from a noise distribution, e.g., a Gaussian distribution.

[0084] At each of multiple denoising steps, the system processes the representation of the set of video frames and the control images and the reference images using the temporal diffusion neural network to generate a prediction of the “clean” or “denoised” set of video frames. The system then uses the prediction to update the representation, e.g., by applying a diffusion sampler, e.g., DDPM, DDIM, or another appropriate sampler, to the prediction and the representation.

[0085] After the final denoising step, the system can use the prediction of the video frames as the final set of video frames.

[0086] In some implementations, the system generates all of the video frames in the output video generated by the video temporal diffusion neural network at once. In some other implementations, the system divides the video frames into subsets, and generates one subset at a time.

[0087] For example, the system can process the initial image and at least a first subset of the control images using the video generation neural network to generate a first subset of video frames from the video. The system can then process one or more of the video frames from the first subset and at least a second subset of the control images using the video generation neural network to generate a second subset of video frames from the video. The system can continue performing this process until all of the video frames in the video are generated. As a particular example of this, the system can use this technique to perform iterative temporal outpainting to extend the length of the generated video. When performing temporal outpainting, the system first generates N frames, and then iteratively outpaints N' < N frames based on the previous N - N'. The amount of overlap between two consecutive clips, i.e. N-N', is chosen as a trade-off between quality and running time.

[0088] Each super resolution neural network performs super resolution on a respective input video for the super resolution neural network, generates an output video that has a higher resolution than the input video for the super resolution neural network.

[0089] Thus, the input video for the first super resolution neural network in the sequence is generated by the temporal image diffusion neural network and the input video input video for any subsequent super resolution neural network in the sequence is generated by the preceding super resolution neural network in the sequence.

[0090] Generally, the super resolution neural network(s) in the sequence can have any appropriate architecture that allows the neural network to increase the resolution of an input video.

[0091] For example, the super resolution neural networks in the sequence can each be a respective image diffusion neural network. In the specific example 400, the super resolution neural networks in the sequence are each respective temporal image diffusion neural networks.

[0092] The video generation neural network can be trained using any of a variety of techniques. For example, after the motion generation neural network has been trained, the video generation neural network can be trained on the same data set of videos with audio or a different data set. In particular, during training, each training example can include (i) one input frame from a corresponding training video, (ii) control images generated from the corresponding training video by applying the motion generation neural network to the corresponding conditioning input, and (iii) a set of target frames that follow the input frame in the corresponding training video.

[0093] Alternatively, during training, each training example can include (i) one input frame from a corresponding training video, (ii) control images generated from the corresponding training video based on the target motion controls for the training video generated as described above, and (iii) a set of target frames that follow the input frame in the corresponding training video.

[0094] In some examples, the video generation neural network is trained using a diffusionbased approach, e.g., an approach in which noise e1is added to the ground truth images I, with the model trained to predict the added noise. An example of a loss function that can be used for such training is: where C is the set of masks (e g., semantic, sparse and / or dense masks), xo‘ is an input image, t is the timestep, and is the predicted noise. Other ty pes of diffusion loss may alternatively be used.

[0095] FIG. 5 shows examples 500 of videos generated using the described techniques. In particular, each row of images represents a video that was generated based on the image shown in the first column and a corresponding sample audio input. The resulting videos are photorealistic and coherent. As can be seen in the synthesized images in the right columns, the described techniques generate head motion, gaze, blinking, lip movement and, unlike previous methods, upper-body and hand gestures.

[0096] For information regarding responsible use of the described techniques, please see the inventors' writing on the subject, e g., those available at arXiv:2403.08764 or at enri ccorona. github .io.

[0097] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0098] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0099] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0100] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0101] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0102] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory' can be supplemented by. or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0103] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory’ devices, including by way of example semiconductor memory devices, e.g., EPROM. EEPROM, and flash memory' devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0104] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0105] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0106] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0107] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g.. an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area netw ork (LAN) and a w ide area network (WAN), e.g., the Internet.

[0108] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e g., a result of the user interaction, can be received at the server from the device.

[0109] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0110] Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0111] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0112] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: obtaining an initial image of an agent; obtaining a conditioning input characterizing sound emitted by the agent; and generating, from the initial image and the conditioning input, a video spanning a first time period of the agent emitting the sound, comprising: processing the conditioning input using a motion generation neural network to generate motion controls for the agent; generating, from the initial image of the agent and the motion controls, a sequence of control images of the agent that characterize motion of the agent during the first time period; and processing the initial image and the sequence of control images using a video generation neural network to generate the video of the agent emitting the sound.

2. The method of claim 1, wherein the conditioning input characterizes speech of the agent.

3. The method of claim 1 or 2, wherein the conditioning input comprises an audio signal.

4. The method of claim 1 or 2, wherein the conditioning input comprises a text transcription of an audio signal.

5. The method of any preceding claim, wherein the video generation neural network comprises a temporal image diffusion neural network.

6. The method of claim 5, wherein the temporal image diffusion neural network comprises: a first branch of layer blocks that for each of a plurality of time points in the video, is configured to receive a first input comprising a current representation of a video frame at the time point in the video and to process the first input to generate a denoising output for updating the current representation; a second branch of layer blocks that, for each of the plurality of time points in the video, is configured to receive a second input comprising the respective control images corresponding to the time point and to process the second input to generate a respective layer output for each layer block in the second branch, wherein: one or more layer blocks in the second branch include a temporal layer that, for each time point, receives inputs from (i) a corresponding layer block in the first branch, (ii) a corresponding layer block in the second branch, or (iii) or both for one or more other time points; and one or more layer blocks in the first branch receive input from a corresponding layer block in the second branch.

7. The method of claim 6, wherein the second input comprises the current representation of the video frame at the time point.

8. The method of any one of claims 5-7, wherein the video generation neural netw ork further comprises a sequence of one or more super resolution neural networks that each perform super resolution on a respective input video for the super resolution neural network, wherein the input video for the first super resolution neural network in the sequence is generated by the temporal image diffusion neural netw ork and the input video for any subsequent super resolution neural networks in the sequence is generated by a preceding super resolution neural network in the sequence.

9. The method of claim 8, wherein each super resolution neural netw ork is a respective image diffusion neural network.

10. The method of claim 9. wherein each super resolution neural network is a respective temporal image diffusion neural network.

11. The method of any preceding claim, wherein processing the initial image and the sequence of control images using a video generation neural network to generate the video of the agent emitting the sound comprises: processing the initial image and at least a first subset of the control images using the video generation neural network to generate a first subset of video frames from the video; and processing one or more of the video frames from the first subset and at least a second subset of the control images using the video generation neural network to generate a second subset of video frames from the video.

12. The method of any preceding claim, wherein the motion controls for the agent comprise, for each of a plurality of time points during the first time period, a first set of controls defining facial expression parameters of a face of the agent at the time point.

13. The method of any preceding claim, wherein the motion controls for the agent comprise, for each of a plurality of time points during the first time period, a second set of controls defining head pose parameters of a head of the agent at the time point.

14. The method of any preceding claim, wherein the motion controls for the agent comprise, for each of a plurality of time points during the first time period, a third set of controls defining body pose parameters of a body of the agent at the time point.

15. The method of any preceding claim, wherein the control images comprise, at each of a pl ural ity of time points during the first time period, a control image characterizing a pose of the agent at the time point.

16. The method of any preceding claim, wherein the control images comprise, at each of a plurality of time points during the first time period, a control image characterizing positions of semantic regions of a head of the agent at the time point.

17. The method of any preceding claim, wherein generating, from the initial image of the agent and the motion controls, a sequence of control images of the agent that characterize motion of the agent during the first time period comprises: applying, to the initial image of the agent, a statistical model to generate parameters of an initial configuration of the agent in the initial image; and applying the motion controls to the initial configuration to generate updated parameters at each of a plurality of time points during the first time point.

18. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any preceding claim.

19. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-17.

Images