Generative neural lip reanimation driven by controllable facial landmarks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for adapting a first video of a speaker speaking a first speech variant characterized by a first lip

Description

GENERATIVE NEURAL LIP REANIMATION DRIVEN BY CONTROLLABLEFACIAL LANDMARKSBACKGROUND

[0001] This specification relates to processing data using machine learning models. Machine learning models receive an input and generate an output, e.g.. a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0002] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0003] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that can generate realistic lip reammation for a video of a speaker. In this specification, lip reanimation refers to editing the lip shape of a speaker in an original input video. In particular, the system can edit the lip shape of the speaker in the original input video such that the speaker appears to be speaking a target audio different than the one that accompanied the original input video.

[0004] More specifically, lip reanimation refers to editing the lip shape of a speaker such that the speaker looks like they are making different sounds, e.g., phonemes, than the speaker made in the original video. In this specification, the lip shapes that result in the original audio and the target audio are referred to as lip movement trajectories, e.g. a first or original lip movement trajectory results in the original audio and a second or target lip movement trajectory results in the target audio.

[0005] The system described can take input video frames from an input video depicting a speaker speaking an original speech variant characterized by an original lip movement trajectory and generate a new frame image of the speaker with the predicted lip shape of a different lip movement traj ectory characterized by the target speech variant. When successive frames are reanimated and combined, the speaker appears to be speaking the target speech variant in a lip reanimated video.

[0006] This specification describes a variety of techniques that can be used to process an input video of a speaker speaking an original speech variant to generate images of the speakerspeaking a target speech variant that can then be combined into a lip reanimated video of the speaker.

[0007] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0008] In an example model discussed in this specification, a generative adversarial network (GAN) is used to generate the output images that can be combined into the lip reanimated video of the speaker speaking the target speech variant. The generator processes inputs characterizing the facial landmark geometry of the lip motion in the target speech variant, a reference head pose image of the speaker, a reference lip image of the speaker, and the temporal context of the speaker's face leading up to the frame to be reanimated to generate the output image. The generator is trained with feedback from a number of discriminators that attempt to distinguish the fake generated output frame from the real frame, i.e. the frame that was not reanimated, such that the generator improves at generating output images of the second speech variant that fool the discriminator.

[0009] The system as specified is speaker-independent, meaning the system can generate an output image for any speaker from inputs that pertain to the speaker. This speakerindependent approach leads to faster turnaround time for lip reanimation, since there is no need to train a new model for every speaker. Additionally, the faster turnaround enables better model scaling, since training can be implemented in an unsupervised manner on a variety of video content featuring a vast number of speakers.

[0010] The system as specified is able to perform lip reanimation even for speakers with short duration appearances, non-frontal head poses, non-uniform lighting, and obstructions around the mouth. These advantages enable the system to generate output frames even for the variety of speaking contexts in real cinematic content.

[0011] This system can be used to translate videos into other languages without jarring lip discrepancy. In particular, the lip reanimation techniques described can be used for crosslingual voice transfer (dubbing) to adjust the speaker’s lip movements in an input video to better match the audio of the target language.

[0012] Additionally, the system as specified is able to support human in the loop correction for users to edit the lip movement trajectory of the lip reanimated video. As an example, dubbing staff can modify the speaker facial landmark geometry input to correct the facial landmark geometry' to a specific lip shape, make a stylistic expressive change, e.g., “more lively", or demonstrate the intended second speech variant motion with their own lips to create a new facial landmark geometry' input.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.DESCRIPTION OF DRAWINGS

[0014] FIG. 1 depicts an overview of a speaker-independent lip rendering system.

[0015] FIG. 2 demonstrates an example facial landmark geometry representation.

[0016] FIG. 3 provides an overview7of example inputs to a generative speaker-independent lip rendering model to generate an output frame image.

[0017] FIG. 4 depicts a speaker-independent lip rendering model with a generative adversarial network architecture and an overview of an example training process.

[0018] FIG. 5 depicts an example expression discriminator to ensure the generated output image has the correct facial landmark geometry.

[0019] FIG. 6 depicts an example sequence discriminator to ensure that the generated output image fits the temporal context of the surrounding frames.

[0020] FIG. 7 depicts an example fusion discriminator to ensure that the generated output image combines the inputs in an expected way consistent with the input video.

[0021] FIGS. 8A, 8B, and 8C demonstrate example methods for achieving seamless lip reanimation transitions for cases when a speaker speaks off camera or is silent before or after speaking.

[0022] FIG. 9 is a flow chart of an example process for generating a lip reanimated video from an input video.

[0023] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0024] FIG. 1 depicts a speaker-independent lip reanimation system 100. The speakerindependent lip reanimation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0025] In the particular example depicted, the speaker-independent lip reanimation system 100 includes a geometry prediction model 130, lip rendering model 140, and an upscaling model 150. In this case, the models 130, 140, 150 are not speaker-specific and can provide an end-to- end system for speaker-independent lip reanimation, i.e., the models of the system can be usedfor lip reanimation broadly across different speakers and do not need to be swapped out by speaker-specific models in order to generate the reanimated video for different speakers.

[0026] This specification focuses on the functionality of and techniques that relate to training the lip rendering model 140. In particular, FIG. 1 depicts the lip rendering model 140 within a broader speaker-independent reanimation architecture including a geometry7prediction model 130 and an upscaling model 150.

[0027] More specifically, the geometry prediction model 130 can generate the facial landmark geometry 135 of the target lip movement trajectory for the lip rendering model 140, which generates the output image 145 of the speaker speaking the target lip movement trajectory7. In particular, the geometry prediction model 130 can process raw audio of the target speech variant 120 and the input video 110 of the speaker to produce the intended facial landmark geometry' 135 for the speaker. In some examples, the geometry prediction model 130 can compute the facial landmark geometry 135 from facial landmark detection of the speaker’s face from a different portion of the input video 110. As an example, the facial landmark geometry model can process audio of a speech in French and a video of a speaker speaking in English to produce the facial landmark geometry 135 consistent with the speaker speaking in French.

[0028] In some cases, the facial landmark geometry7135 is a numeric representation of the three dimensional (3D coordinates) of facial landmarks of the face of a speaker in the input video 110. Facial landmarks are essential attributes of a human face, such as the eyes, nose, and mouth, that can be used to distinguish between different faces. As a further example, the facial landmark geometry7135 can include details such as eye comers or dimples. In this specification, the facial landmark geometry 135 can be used for characterizing the lip shape and expression of the intended frame to be reanimated.

[0029] As an example, the facial landmark geometry 135 can be a three-dimensional representation of the image of the intended frame to be reanimated where the three dimensions correspond with the image width, height, and color channels (e.g., red, blue, and green image channels). As another example, the facial landmark geometry 135 can be a two-dimensional representation, e.g. a matrix, that unambiguously encodes facial landmarks, even for speakers with different head orientations. An example matrix representation will be described in further detail in FIG. 2.

[0030] In this example, the speaker-independent lip rendering model 140 ingests the facial landmark geometry7135 and the input video 110 that features a speaker speaking a first speech variant to generate an output image 145 that makes the speaker look like they are speaking atarget speech variant. In a particular implementation, for each input video frame within the input video 110, the model 140 can process the input video frame, one or more reference images derived from the input video 1 10 and the corresponding facial landmark geometry 135 to generate an output image 145 in line with the intended frame to be reanimated that characterizes the target speech variant. An example model that processes a reference temporal context sequence of images, a reference pose, and reference lip image, will be covered in more detail with respect to FIG. 3.

[0031] The lip rendering model 140 can be any appropriate neural network architecture that can be configured to process a variety of different input ty pes and sizes, as will be described with respect to FIG. 3. In some examples, the model is a generative model that can process a variety of inputs to produce new images, such as the output image 145. More specifically, the generative model can leam the data distribution of the inputs and outputs of a training set to generate new data points drawn from the same distribution. As an example, a variational autoencoder (VAE) or generative adversarial network (GAN) architecture can be used to generate the output image 145. As another example, the lip rendering model 140 can be implemented as a convolutional neural network (CNN) or a vision transformer.

[0032] In this case, the lip rendering model 140 is speaker-independent because it can process the input facial landmark geometry' 135 and input video 110 of any speaker to generate an output image 145 depicting the speaker speaking the target speech variant. To allow the model to generalize to different speakers, the system 100 or another training system can train the lip rendering model 140 as described below with reference to FIG. 4.

[0033] In some cases, the output image 145 produced by the speaker-independent lip rendering model 140 is not the same resolution as the resolution of the speaker input video 110, e.g. the output image 145 can be generated with a lower resolution than the input video 110. In this case, the lip rendering model 140 can provide the output image 145 as an input to an upscaling model 150 that produces the final high-resolution output image 160 by blending the low- resolution output image with the speaker input video 110 of the original resolution.

[0034] In the particular example depicted, the upscaling model 150 is included in the system 100 to process the output image 145 with the original speaker input video 110 in order to generate a high-resolution output image 160. In other cases, the output image 145 of the lip rendering model 140 is already high-resolution and does not need to be upscaled or blended into the input video 110.

[0035] The speaker-independent lip reanimation system 100 can combine a sequence of one or more output images 145 or one or more high-resolution output images 160 with the target audio120 to generate the reanimated video for different speakers. More specifically, each input video frame of the speaker input video 110 can be reanimated, combined sequentially, and incorporated with the target audio 120 to generate an output video of the speaker speaking the lip movement traj ectory of the target speech variant.

[0036] FIG. 2 depicts an example facial landmark geometry' representation (e.g., the facial landmark geometry’ 135 of FIG. 1) to encode the predicted facial landmark coordinates of the speaker’s face in the frame to be reanimated.

[0037] In this example, the facial landmark geometry 135 is encoded as a two-dimensional matrix representation with the same dimensions, e.g. width 202 and height 204 as the input video 110 frame image. Representing the facial landmark geometry 135 in two-dimensions enables the geometry’ 135 to be encoded as a matrix with a single layer, thereby reducing the computational resources necessary for processing inputs to the lip rendering model 140, e.g. with respect to using a three-dimensional representation of the image which requires greater data throughput from the lip rendering model 140.

[0038] More specifically, the single layer two-dimensional matrix representation can encode facial landmark information in a way that unambiguously indicates head orientation, e.g. by filling in non-zero values centered at each landmark’s position within the matrix. In particular, the values of the matrix can indicate which part of the face the landmarks correspond to. Furthermore, the two-dimensional matrix representation can efficiently encode the speaker’s facial landmarks by including only visible landmarks. For example, for a face in profile in which the speaker’s left eye cannot be seen, landmarks pertaining to the left eye can be excluded from the facial landmark geometry.

[0039] As an example, a blank image without a landmark can be represented as a matrix of zeroes, e.g., the no landmark geometry 200. For images that contain facial landmarks, landmarks on the left side of the face can be represented as negative values, e.g., the landmark from the left side of the face geometry’ 210, and landmarks on the right of the face can be represented as positive values, e.g., the landmark from the right side of the face geometry 220. In particular, the center of these landmarks can be defined at -1.0 for a left-sided landmark and +1.0 for an exact center or right-sided landmark.

[0040] The example two-dimensional matrix representation can properly handle cases such as a tilted head, e.g., because landmarks from the left side of the face can appear above landmarks of the right side of the face, or a head in profile, e.g., because landmarks from the left or right side of the face may not be visible in profile. In particular, the facial landmark geometry 135can be reoriented and aligned with the head pose of the speaker in the intended frame to be reanimated.

[0041] In a case in which facial landmark detection is used to create the facial landmark geometry 135, an uncertainty tolerance for the exact facial landmark position can be included within the facial landmark representation. More specifically, the geometry' prediction model (e.g., the geometry prediction model 130 of FIG. 1) can be configured to generate the two- dimensional matrix facial landmark geometry representation and an associated uncertainty matrix. The uncertainty values of the matrix can then be combined with the two-dimensional matrix to scale the detected landmark point centers marked within the two-dimensional matrix representation in accordance with the uncertainty' of the landmark. In particular, the values surrounding a landmark can be increased or decreased within a diameter associated with the uncertainty of landmark detection.

[0042] FIG. 3 provides an overvieyv of hoyv example inputs to a speaker-independent lip rendering model (e.g., the speaker-independent lip rendering model 140 of FIG. 1) can be selected for each input video frame in order to generate a particular output image for each video frame. In particular, for each video frame, the system (e.g., the speaker-independent lip reanimation system 100 of FIG. 1) can choose and combine the facial landmark geometry (e.g., the facial landmark geometry 135 of FIG. 1) with one or more reference image inputs in the same order for processing by the speaker-independent lip rendering model 140 in order to generate the output frame.

[0043] In this particular example, the speaker-independent lip rendering model 140 is the generator 300 of a generative adversarial netyvork (GAN). GANs are trained using the feedback of discriminators to critique their generated output. Example discriminators and a generative speaker-independent lip rendering model training process will be covered in further detail in FIGS. 4-7.

[0044] Beyond the facial landmark geometry 135, example inputs can include reference images 330 taken from the input video 110 or another video containing the speaker. As an example, the reference images 330 can include other images of the speaker that characterize some aspect of the frame to be reanimated, such as an intended facial expression, head orientation, color saturation, presence of obstructions, and teeth and tongue positioning. In particular, the reference images 330 can include a reference head pose 332 to provide the head orientation, saturation, and presence of obstructions and a reference lip image 334 to provide the intended facial expression and teeth and tongue positioning.

[0045] In the case that the reference image inputs 330 include a reference head pose 332 and reference lip image 334, the reference head pose 332 can characterize the head pose, lighting, and transient visual conditions of the speaker in the frame to be reanimated. As an example, transient visual conditions can include obstructions, shadows, a change of lipstick, milk mustaches, the effect of video fade in or out on the level of facial detail, or food residue on the speaker mouth. The reference lip image 334 can characterize the lip geometry of the speaker in the frame to be reanimated as well as the positioning of the teeth and tongue.

[0046] In some examples, the system 100 can select the reference pose image 332 from the frame to be reanimated, since the frame to be reanimated has the exact head pose, lighting, and transient visual conditions that characterize the intended output image 145. In the case of the speaker moving around in the input video 110. the reference pose image 332 can also be chosen from a consecutive series showing consistent head pose motion in consecutive frames. The reference lip image 334 can be chosen relative to the intended characteristics of the face in the frame to be reanimated and can be taken from any available video containing the speaker, including the input video 110.

[0047] In another example, the system 100 can select the reference lip image 334 from a frame in which the facial landmark geometry 135 associated with the lips closely matches the facial landmarks that pertain to the lips of the frame to be reanimated. Additionally, the reference lip image 334 can be chosen such that the head pose, lighting, and transient visual conditions are similar to the head pose, lighting, and transient visual conditions of the identified reference pose image 332. In particular, a reference lip image 334 can be chosen from a frame nearby in time to the frame to be reanimated.

[0048] In yet another example, the reference lip image 334 can be chosen with a different head pose and lighting than the identified reference pose image 332. In particular, this alternate choice can be done when training the generator 300 to encourage the generator 300 to leam how to transform between different reference poses 332 and reference lip images 334 with different poses and lighting conditions. In particular, this alternate choice can be made for some fraction of the input data in training, such as 10% or 25% of the time, to not make learning too difficult.

[0049] In some cases, it may not be possible to find a reference lip image 334 that exactly characterizes the target lip shape of the target speech variant. In this case, the inclusion of the facial landmark geometry' 135 can address any discrepancy between the available examples of reference lip shape 334 and the target lip shape of the frame to be reanimated.

[0050] As a further example, the reference images 330 can include a temporal context sequence of images 310 to temporally contextualize characteristics of the speaker in the frames that surround the frame to be reanimated. In particular, a sequence of N context images 310 that temporally situate the frame to be reanimated can be incorporated into the generator 300 input. The temporal context sequence 310 ensures that the motion created from combining the generated output frame images 145 is temporally consistent and does not produce any confounding effects such as facial landmarks appearing in slightly different places or in slightly different colors. In some cases, the temporal context input sequence 310 can include N prior consecutively generated or combined faces leading up to the frame to be reanimated. In other cases, the temporal context input sequence 310 can include N prior ordered, e.g., nonconsecutive, frames leading up to the frame to be reanimated. As an example, N can be a number, such as 1, 8, or 20 contextualizing frames.

[0051] In some examples, the number N of context images can be chosen with respect to the frame rate of the input video 110, e.g. the temporal context sequence of images 310 can be chosen to cover a certain amount of elapsed video time. More specifically, in the case of a 30 frames per second (fps) input video 110, the system can use an N between 1 and 8 to temporally contextualize the frame to be reanimated with a time duration of 1 / 30 to 8 / 30 of a second. As an example, in the case of a 90 or 120 fps input video 110, the system can use an N between 3 and 24 or between 4 and 32 to provide the same elapsed video time context. In another example, the temporal context sequence of images 310 can be chosen at 30-40 ms offsets, e.g. skipping intermediate frames, such that the number N can cover more elapsed video time by incorporating nonconsecutive but still ordered images leading up to the frame to be reanimated.

[0052] As another example, in the case of a continuous speaking shot, the previous N frames of the generated output images that characterize the target speech can be combined into the temporal context images 310. In the case where there is a non-continuous shot change of the speaker, such as a jump-cut, a process for combining faces for seamless transitioning can be used to generate the temporal context sequence 310. Techniques for seamless transitioning will be covered in further detail in FIGS. 8A-8C.

[0053] As an example, the facial landmark geometry 135, temporal context images 310, and reference images 330 can be concatenated, and processed by the generator 300. More specifically, the generator can be a neural network model, such as a CNN, autoencoder, or vision transformer that is configured to process multiple different sized outputs.

[0054] For instance, in some cases, the inputs 135, 310, and 330 can be flattened into a single input array before being processed by the generator neural network. In other cases, zeropadding can be used to fill in the variable input sizes of the inputs 135, 310, and 330 to match the maximal input size and a subnetwork can be defined within the generator neural network to process each input sequentially in order to learn a representation of the group of inputs.

[0055] As another example, convolutional layers can be combined with spatial pyramidal pooling in a CNN to process regions of each input image into a fixed-length representation. In yet another example, a vision transformer can sequentially process and divide each input into fixed-size patches and the patches can be embedded into fixed-length input embeddings.

[0056] The generator 300 processes the one or more inputs to generate an output image 145 of the speaker speaking the target speech variant. This output 145 then replaces the frame to be reanimated. In this example, the output 350 generated is not the whole speaker face. In particular, a cropping scheme 360 can be adopted such that the generator 300 can only generate a portion of the mouth area for the frame to be reanimated as the output image 145. More specifically, the generator 300 can be configured to modify the lip portion of the image instead of the whole face to ensure that the generator 300 can be relatively computationally efficient with respect to the input data, thereby reducing the use of computational resources. More specifically, the cropping scheme 360 can specify that the output image 350 can be cropped more than the one or more input images during training such that the generator 300 can learn to output a smaller region of the image with respect to the input image.

[0057] In particular, a cropping scheme 360 can designate cropping the output image in a way that targets the mouth area portion of the input image while cropping the input image in a less restrictive way. As an example, the cropping scheme 360 used in the training of the generator 300 can specify for the one or more input reference images 330 to be cropped vertically from mid-forehead to the base of the neck and horizontally from ear to ear in an input image crop 362; and can specify for the output image 145 to be cropped vertically from mid-nose to midneck and horizontally from outer cheek to outer cheek in an output image crop 364. More specifically, the cropping scheme 360 can provide the generator 300 with more information about how the speaker’s face is aligned and oriented in a wider shot that contains landmarks like the eyes and ears, while restricting the generator’s 300 ability to modify only the mouth region of the face such that it can be prevented from making unnecessary changes to the speaker’s whole face.

[0058] In the case of the output image 350 of the generator 300 being the mouth area portion of the image, the output image 350 can be recombined with the frame to be reanimated by aligning the mouth area portion with respect to the corresponding aspect ratio, location, and size of the frame to be reanimated, e.g., the mouth area portion output image can be placedback in exactly the mouth area location of the frame to be reanimated. In some cases, the mouth area portion output image 350 can be blended in with the frame to be reanimated. As an example, a mask of semi-translucent pixels can be applied around the borders of the output image to avoid any abrupt visual discontinuities when recombining the output image 350 back with the frame to be reanimated.

[0059] FIG. 4 depicts an overview of a GAN model training process in which the generative speaker-independent lip rendering model receives feedback from one or more discriminators that are defined in accordance with one or more discriminator tasks.

[0060] During training, the generator 300 can process the one or more inputs, e.g. the facial landmark geometry' 135 and reference image 330 inputs described in FIG. 3, to generate an output image 145 that can be evaluated by one or more discriminators 410. In particular, the discriminators 410 evaluate the output image 145 and provide feedback 420 to the generator 300 about how effectively the output image 145 fooled the discriminators 410. More specifically, “fooling” the discriminators refers to the discriminators being unable to distinguish the “fake” generated output image 145 from a “real” image. If the generator 300 is able to fool the discriminators, then the generator has succeeded in generating a realistic- looking lip reanimated output image 145.

[0061] In the training process depicted, each discriminator is provided with one or more fake generated images and one or more corresponding real images in accordance with a discrimination task defined for each discriminator to discriminate between the fake and real images. In particular, the fake images referred to in the following description can include the “fake” generated reanimated output image 145 and real, i.e., non-reanimated output images that serve as the fake inputs to the discriminator to provide the appropriate context for evaluating the generator 300. Example discriminator tasks will be covered in further detail in FIGS. 5-7. The discriminator training process involves training each discriminator to distinguish the real images from the fake generated images. More specifically, each discriminator can output a probability value in accordance with whether the real and fake images are drawn from the same distribution or a very similar distribution, e.g., each discriminator quantifies how similar the distribution that the generator leams is to the real underlying distribution of speaker images.

[0062] In an example, the full GAN including the generator 300 and the discriminators 410 can be trained together at the same time using an adversarial or a self-consistent cooperative training method. Over the course of training, the feedback 420 that the discriminators 410 provide to the generator 300 can promote convergence between the generator’s 300 learneddistribution and the actual distribution of real speaker images. More specifically, the feedback 420 defines an objective function for training the generator 300 that can be used to define a loss that quantifies the distance between the distribution of the output images generated by the generator 300 and the distribution of real images. This loss can be backpropagated through the generator 300 and minimized to increase the convergence between the learned distribution and the real distribution.

[0063] The one or more discriminators 410 can be chosen to define tasks that can be strategically combined such that the generator 300 cannot confound all of the discriminators at the same time by copying over information from the inputs 105 and 110 without making any changes. In some examples, the one or more discriminators 410 can include one or more of: a sequence discriminator 412 that defines a temporal consistency discrimination task, an expression discriminator 414 that defines a lip geometry discrimination task, and a fusion discriminator 416 that defines a lip geometry within the broader context of the whole speaker face discriminator task. Examples of an expression discriminator, a sequence discriminator, and a fusion discriminator will be covered in more detail in FIG. 5-7, respectively.

[0064] The feedback provided by the discriminators 410 can take a variety of forms. For example, a weighted sum of the one or more discriminator scores can be used as combined feedback for the generator model. In particular, the objective function for training the generator can be a combined function of the one or more discriminator probability outputs.|00065| In another example, the feedback 420 provided by the one or more discriminators 410 to the generator 300 can evolve throughout the training process. In particular, the feedback 420 can be based only on the best scoring discriminator, the discriminator that is the most effective critic of the generator, i.e. the discriminator that is most confident in identifying a fake input when that input comes from the generator 300. In this case, the generator 300 can be forced to continually improve to address the most critical discriminator throughout training, which ensures that the generator 300 achieves acceptable performance on each of the tasks defined by the one or more discriminators 410.

[0066] For example, in the particular case with the expression, sequence, and fusion discriminators 412, 414, and 416 described above, if the generated output image 145 was merely copied over from the inputs in such a way that it fooled two of the three discriminators and the feedback 420 implemented an equal-weighted feedback strategy7, the generator’s score for cheating can be 67%, making it prohibitive for the generator 300 to consider an alternate uncopied solution. However, in the case of evolving feedback, the generator can improve withregard to the best scoring discriminator, e.g., the unfooled discriminator, and cannot succeed in fooling the discriminators by merely copying over the inputs in order to improve.

[0067] FIG. 5 depicts an example expression discriminator configured to ensure the predicted mouth area of the output image has the correct lip geometry. The correct lip geometry can be defined to be consistent with the facial landmark coordinate geometry' that characterizes the target lip geometry of the frame to be reanimated.

[0068] The expression discriminator 412 can process the real or fake mouth area image 500 and the corresponding facial landmark geometry 510 as input. In the particular example shown, the discriminator 412 can process a cropped view 500 of the output image 145 and the facial landmark geometry' 510. In the case where the generator 300 generates a cropped mouth area image directly, the discriminator 412 can process the already cropped output and can crop the facial landmark geometry 510 accordingly.

[0069] As an example, the real input image can include the mouth area from the frame to be reanimated and the corresponding real facial landmark geometry can be computed from the full view of the frame to be reanimated. As another example, a fake input face can include the mouth area as generated by the generator 300 and the corresponding facial landmark geometry of the speaker’s face from a different portion of a speaker video.

[0070] The expression discriminator 412 can be trained to process the respective real or fake inputted mouth area 500 and facial landmark geometry 510 to predict the expression discriminator output 550 defining a probability that the mouth area has the correct geometry, i.e. that the mouth area of the fake image matches the facial landmark geometry.

[0071] FIG. 6 depicts an example sequence discriminator configured to ensure that the predicted output image in sequence with temporal context frames, e.g., prior consecutive or ordered nonconsecutive frames, is temporally consistent with the natural series of motion defined by the target speech variant.

[0072] In this case, the sequence discriminator 414 processes the temporal context input, e.g., the temporal context 310, with the real or fake next frame 600, e.g. the real frame to be reanimated or the fake reanimated output image that was reanimated by the generator 300, to assess whether combining the series of images results in a temporally consistent series of consecutive images. As an example, a real input sequence can include the prior N consecutive or ordered nonconsecutive images leading up to the frame to be reanimated, e.g., the temporal context input 310, combined with the frame to be reanimated. As another example, a fake input sequence can include the same temporal context input 310 to the generator 300 and the reanimated image output 145 generated by the generator 300. The sequence discriminator 414can be trained to process the inputted temporal context 310 and the respective real frame to be reanimated or the fake output image 600 to predict the sequence discriminator output 650 defining a probability that the sequence is real.

[0073] In particular, the sequence discriminator 414 can process a sequence of images featuring the full uncropped view of the real or fake face. In the case where the generator generates a cropped mouth area image as output, the cropped output 350 can be combined with the reference pose image 332 to provide the full fake next frame image 600. In particular, configuring the sequence discriminator 414 to process inputs with a wider view than what the generator 300 generates can promote the detection of inter- and intra-frame spatial inconsistencies. More specifically, the sequence discriminator 414 can detect intra-frame (spatial) and inter-frame (temporal) visual discrepancies such as noise, detail drift, or random color fluctuations. For example, the sequence discriminator 414 can distinguish if the speaker’s teeth are changing shape from frame to frame or if the mouth area is inappropriately lighter or darker than the surrounding face.

[0074] In certain examples, utterance starts, e.g. frames when a speaker starts speaking after not speaking, can be excluded from being input into the sequence discriminator 414. Utterance start exceptions do not have any prior frames with the speaker speaking to use as the temporal context sequence 310. In this case, the N faces of the temporal context sequence 310 provided to the generator 300 can be repeated faces of a nearby frame. If included, these identical frames can confound the sequence discriminator 414 and encourage the generator 300 to generate an identical output frame, which is not a desired result.

[0075] FIG. 7 depicts an example fusion discriminator configured to ensure that the attributes of the input images and facial landmark geometry are combined in the expected way.

[0076] In particular, the fusion discriminator 416 ensures that the face of the output image 145 is consistent with the lip geometry, orientation, lighting, and transient visual conditions of the frame to be reanimated. More specifically, the fusion discriminator ensures the generated output image 145 has both the expected lip shape and is similarly detailed and photorealistic as the mouth area in the original input video of the speaker 110 using one or more real or fake reference images.

[0077] In the particular example depicted, the fusion discriminator 416 can process the real or fake mouth area 500 and corresponding cropped real or fake reference images, e.g., real and fake reference pose 732 and lips 734 images. Processing the cropped view can ensure that the larger image context is excluded from the fusion discriminator 416 input, especially the eyes and mouth, e.g., since similarities in those areas between the real and fake images can makethe discrimination task too easy. Thus, using the cropped view can ensure that the fusion discriminator 416 is able to provide well-informed feedback without an unfair advantage. In particular, this can promote the generator 300 learning to synthesize faces that look like they belong in the video, i.e. that the head pose, lighting, transient visual conditions, and lip shape are consistent with the input video.

[0078] As an example, the real mouth area can come from cropping the frame to be reanimated and the fake mouth area can come either directly from or by cropping the output 145 of the generator 300. In the case where the generator 300 generates a cropped view image as an output image 350, the discriminator 416 can process the cropped mouth area directly as the fake mouth area.

[0079] The real or fake reference images 730 can correspond with the same types of reference images 430 included in the generator input. In the particular example depicted, the reference images can include real or fake reference pose 732 and reference lip 734 images. More specifically, the reference head pose 732 can provide the head orientation, saturation, and presence of obstructions and the reference lip image 734 can provide the intended facial expression and teeth and tongue positioning. If the fake generated mouth area lacks the detail it should have, e.g., blurred teeth, or has too much detail, e.g., an inappropriately crisp image during a video fade-out, this will be made obvious to the fusion discriminator 416 in comparison to the reference images 732 and 734.|00080| For desired functioning of the fusion discriminator 416, it is important to ensure that the real mouth area, reference pose, and reference lip images and the fake mouth area, reference pose, and reference lip images cannot be the same. In the case that the reference images correspond with the same types of reference images 430 input to the generator 300, the fake reference pose and reference lip images can be the reference pose 332 and reference lip 334 images input into the generator 300 since the fake mouth area input is generated by the generator 300 and is different than the reference pose 332 and reference lip 334 images. However, the real reference pose or reference lip images cannot come from the frame to be reanimated, which is being used as the real mouth area input, since this would enable the fusion discriminator to exploit the fact that either the reference pose, reference lip, or both, and the real mouth area images would all be the same or come from the same image.

[0081] In particular, the real reference head pose can be chosen from a nearby frame to closely match the frame to be reanimated head pose, lighting, and transient visual conditions. If no other face can be found that has the right head pose, the head pose of some other frame can be warped to match the head pose of the frame to be reanimated. Furthermore, the real referencelip image 734 can be chosen using the facial landmark geometry 135 of the frame to be reanimated to find another frame that has a highly similar lip shape to the frame to be reanimated. As an example, choosing a real reference lip image 734 with a good lip match but a fairly different head pose may help in training the generator 300 by encouraging the generator 300 to work with different head poses and different lighting. In particular, these alternate choices can be made for some fraction of the input data, such as 10% or 25% of the time, to not make learning too difficult in training.

[0082] The fusion discriminator 416 can be trained to process the real or fake mouth area 500 and corresponding real or fake reference images 732 and 734 to predict the fusion discriminator output 750 defining a probability that the face belongs within the original input video 110.

[0083] FIGS. 8A, 8B, and 8C demonstrate examples of generating temporal context frames, e.g. the temporal context frame 310 of FIG. 3, for achieving seamless transitioning of lip reanimation for three cases in which a speaker speaking is preceded or postceded by the speaker either not speaking or speaking out of the frame. These methods can be used to make the transition from seeing the speaker not yet speaking to speaking or not yet speaking to speaking look natural, e.g. to ensure that there is no abrupt jerk in the lip shape and the level of detail from one frame to the next when the speaker starts or stops speaking in the reanimated sequence.

[0084] In the case shown in FIG. 8 A, there are not enough prior frames with the speaker’s face leading up to the first frame to reanimate. In particular, the scene can feature a discontinuous jump shot and the speaker can have been speaking off camera before the camera pans over to the speaker.

[0085] In the example method shown, the face from the first adjacent frame in the shot within the face can be included in place of each missing face frame to create the temporal context sequence 310. In particular, the chosen face can be repeated for the N frames that parametrize the temporal context sequence.

[0086] In the case shown in FIG. 8B, there are enough prior frames with the speaker’s face leading up to the first frame to reanimate, but the speaker is not yet speaking in the prior frames.

[0087] In the example method shown, k leading frames in addition to the speaking frames can be reanimated to ensure a smooth transition from the speaker’s lip shapes in those frames to the frames with speech that have been selected for reanimation. The leading frames can be reanimated by interpolating the lip geometry using past face image context for the generator’s temporal context image input sequence. In particular, the lip geometries of the first speakingframe i and the i-k (the kth frame before i) can be used to interpolate a motion in k preceding frames to ensure the reanimation of the intended frames looks natural.

[0088] In the case shown in FIG. 8C, there are later frames with the speaker’s face following the last frame of an utterance in which the speaker is no longer speaking.

[0089] In the example method shown, k following frames in addition to the speaking frames can be reanimated to ensure a smooth transition from the reanimated lip shapes to the speaker’s lip shapes in the unspeaking frames. The following frames can be reanimated by interpolating the lip geometry using past face image context for the generator’s temporal context image input sequence. In particular, the lip geometries of the first speaking frame and the i+k (the kth frame after i) can be used to interpolate a motion in k successive frames to ensure the reanimation of the intended frames looks natural.

[0090] FIG. 9 is a flow chart of an example process for generating a lip reanimated video from an input video. For convenience, the process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed in accordance with this specification, such as the speakerindependent lip reanimation system 100 of FIG. 1, can perform the process 900.

[0091] The system can receive an input video of a speaker for reanimation purposes in which the speaker is speaking a first speech variant and needs to be made to appear to be speaking a second speech variant (step 910). In particular, the first speech variant can be a speaker’s original speech and the second speech variant can be a target speech for the speaker to speak. In some cases, the second speech variant can be speech in a different language. For each input video frame in the input video of the speaker, the system can generate a predicted facial landmark coordinate geometry characterizing a second lip movement trajectory of the second speech variant (step 920).

[0092] The system can then identify and combine reference images for the output image characterizing the second lip movement trajectory (step 930), and process the predicted facial landmark coordinate geometry and the reference images for the frame to be reanimated (step 940) to generate an output image corresponding with the second lip movement trajectory’ (950). In some examples, the reference images can include a temporal context sequence as well as reference pose and lip images to contextual the frame to be reanimated.

[0093] The system repeats steps 920-950 for each input video frame in the input video to generate a sequence of output image frame images that can be combined into a video of the speaker speaking the second speech variant (step 960). In the case of the second speech variantbeing speech in another language, the combined lip reanimated frames make the speaker's lips follow a lip movement trajectory consistent with the speech in the other language.

[0094] This specification uses the term ‘'configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0095] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e.. one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0096] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0097] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can bewritten in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0098] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0099] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.[000100] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.[000101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.[000102] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.1000103] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. [000104] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Jax framework.[000105] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a frontend component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.[000106] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.[000107] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a subcombination or variation of a subcombination.1000108] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order show n or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.[000109] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

Claims

CLAIMS1. A method comprising: receiving an input video frame from an input video that depicts a speaker speaking a first speech variant, the first speech variant characterized by a first lip movement trajectory’; generating a predicted facial landmark coordinate geometry of a second speech variant characterized by a second lip movement trajectory’; obtaining a plurality’ of reference images for the input video frame; and generating an output video frame of the speaker with a target lip geometry consistent with the second lip movement trajectory based on the predicted facial landmark coordinate geometry and the plurality of reference images using a generator neural network.

2. The method of claim 1, further comprising: receiving a second input video frame from a second input video that depicts a second speaker speaking a third speech variant, the third speech variant characterized by a third lip movement traj ectory; generating a predicted facial landmark coordinate geometry of a fourth speech variant characterized by a fourth lip movement trajectory; obtaining a plurality of reference images for the second input video frame; and generating an output video frame of the speaker with a target lip geometry consistent with the fourth lip movement trajectory’ based on the predicted facial landmark coordinate geometry of the fourth speech variant and the plurality' of reference images for the second input video frame using the generator neural network.

3. The method of any one of claims 1-2, wherein the second speaker is the same speaker as the speaker.

4. The method of any one of claims 1-3, wherein the second speech variant is a translation of the first speech variant.

5. The method of any one of claims 1-4, wherein the output video frame of the speaker is cropped more than the input video frame.

6. The method of any one of claims 1-5, wherein obtaining the plurality of reference images for the input video frame comprises: obtaining a reference pose image that characterizes a target head pose of the speaker in the input video frame; obtaining a reference lip image of the speaker that characterizes the target lip geometry of the speaker in the input video frame; and obtaining a sequence of N context images from a sequence of N adjacent video frames to the input video frame that temporally contextualize the input video frame.

7. The method of claim 6, wherein, due to being conditioned on the predicted facial landmark coordinate geometry, the output video frame addresses one or more discrepancies between the reference lip image of the speaker and the target lip geometry.

8. The method of any one of claims 1-7, wherein the predicted facial landmark coordinate geometry is encoded as a two-dimensional matrix of the same width and height as the input video frame and comprises a geometric representation of a plurality of visible speaker facial landmarks.

9. The method of claim 8, wherein encoding the geometric representation of a plurality of visible speaker facial landmarks comprises: assigning a plurality of negative values to represent a plurality of visible speaker leftside facial landmarks; and assigning a plurality of positive values to represent a plurality of visible speaker rightside facial landmarks.

10. The method of claim 9, wherein a number of of assigned positive or negative values around a center point that represents a center of a particular visible speaker facial landmark indicates an uncertainty of the particular visible speaker facial landmark.

11. The method of any preceding claim when dependent on claim 6, wherein obtaining the sequence of N context images when the sequence of N adjacent video frames to the input video frame do not depict the speaker speaking comprises: repeating a first adjacent video frame N times; orreanimating a sequence of K leading or following video frames adjacent to the input video frame by interpolating a lip movement trajectory in the sequence of K leading or following video frames adjacent to the input video frame.

12. The method of any one of claims 1-11, wherein the generator neural network has been trained adversarially with a plurality of discriminators that each generate a respective discrimination prediction as generator feedback to characterize whether a plurality of respective discriminator inputs comprising an input video frame represent real or fake discriminator inputs, wherein the real discriminator inputs are consistent with the input video and the fake discriminator inputs are not consistent with the input video.

13. The method of claim 12, when dependent on claim 6, wherein the plurality of discriminators comprises two or more of: an expression discriminator configured to process a plurality' of real or fake expression discriminator inputs comprising an input video frame to generate an expression discrimination prediction defining a probability that the input video frame is characterized by the target lip geometry- as generator feedback; a fusion discriminator configured to process a plurality' of real or fake fusion discriminator inputs comprising an input video frame to generate a fusion discrimination prediction defining the probability that the input video frame is consistent with the reference images as generator feedback; or a sequence discriminator configured to process a plurality of real or fake sequence discriminator inputs comprising an input video frame to generate a sequence discrimination prediction defining the probability that the input video frame is consistent yvith the sequence of N context images as generator feedback.

14. The method of claim 13, wherein: the plurality of fake expression discriminator inputs comprises the generated output video frame and a fake facial landmark coordinate geometry computed from facial landmark detection of the speaker from a different video frame of the input video; and the plurality of real expression discriminator inputs comprises the input video frame and a real facial landmark coordinate geometry computed from facial landmark detection of the speaker from the input video frame.

15. The method of any one of claims 13-14, wherein: the plurality of fake fusion discriminator inputs comprises the generated output video frame and the fake reference pose and fake reference lip images comprise the reference pose and reference lip images used by the generator to generate the output video frame; and the plurality' of real fusion discriminator inputs comprises the input video frame and the real reference pose and real reference lip images comprise reference pose and reference lip images from a video frame nearby to the input video frame or a warped frame that closely matches the head pose and lip geometry of the input video frame.

16. The method of any one of claims 13-15, wherein: the plurality of fake sequence discriminator inputs comprises the generated output video frame and the sequence of N context images; and the plurality of real sequence discriminator inputs comprises the input video frame and the sequence of N context images.

17. The method of any one of claims 13-16, wherein the generated output video frame and input video frame processed by the expression and fusion discriminators comprises a real or fake mouth area image cropped around the mouth area of the speaker in the generated output or input video frame.

18. The method of any one of claims 12-17, wherein a best scoring discriminator provides generator feedback to the generator neural network during the adversarial training.

19. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of claims 1-18.

20. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of claims 1-18.

Images