Match lip movements and gestures in digital video to alternative audio

Abstract

A method for matching lip shapes and movements in a digital video with substitute audio includes deriving a sequence of facial poses including lip shapes of an actor from a source digital video. Each pose in the sequence of facial poses corresponds to a middle position of each audio sample. The method further includes generating an animated facial mesh based on the sequence of facial poses and the source digital video, transferring tracked expressions from the animated facial mesh or a target video to the source video, and generating a transferred rough output video including the tracked expressions. The method further includes generating a finished video by refining the rough video at least in part using a parameterized autoencoder trained on lip shapes in the animated facial mesh or the target video. One or more computers may perform the operations of the method.

Description

Technical Field

[0001] The present application relates to digital image processing, and more particularly, to a system or method for matching lip movements and gestures in digital video with substitute audio. Background Art

[0002] Producers of video content often wish to use recorded audio that does not match the images recorded on film. This can occur for several reasons, including, for example, when dialogue is altered after video capture for creative or legal reasons, or when dialogue is recorded in a different language than the actors spoke during a performance for distribution to different audiences. However, consumers of video content often notice that mouth movements or lip movements do not match the timing of the spoken dialogue. Any mismatch in these qualities can distract consumers, reduce the suspension of disbelief, and diminish their enjoyment of the video content.

[0003] Each frame of recorded video can be altered after video capture so that an actor's mouth movements better align with the accompanying speech and dialogue in the audio track. However, methods for altering recorded video are too time-consuming and expensive to be feasible in most releases, reducing the appeal, reach, and profitability of films and other video content that contains dialogue that does not match the actor's mouth movements and associated facial movements. No previous method has been able to achieve high enough quality for distribution of films or other commercial video content at a viable cost. Due to the diversity of image capture parameters used in actors and commercial videos, the material used to train neural networks and other machine learning algorithms is rarely available in sufficient quantity to eliminate obvious mismatches between dialogue and mouth movements, and obtaining sufficient training video can be expensive or impossible.

[0004] It would therefore be desirable to develop new methods and other techniques for matching lip movements and motions in digital video to alternative audio after the initial capture of the images and sounds, thereby overcoming these and other limitations of the prior art. Summary of the Invention

[0005] The present disclosure and the following detailed description should be construed as supplementary to the overall disclosure and may include redundant and / or supplementary topics. The omission of any section does not indicate the priority or relative importance of any element described in the overall application. As is apparent from the individual disclosures, differences between sections may include supplementary disclosures of alternative embodiments, additional details, or alternative descriptions of the same embodiment using different terminology.

[0006] In one aspect of the present invention, a method for matching lip movements and mouth shapes in a digital video to substitute audio includes deriving a sequence of facial poses, including lip shapes of an actor, from a source digital video in which the actor appears. Each shape in the sequence of shapes corresponds to the middle position of each sample of the substitute audio. The method further includes generating an animated facial mesh based on the sequence of facial poses and the source digital video. Optionally, the method may include generating a target video by rendering the animated facial mesh. The method further includes transferring a tracked expression from at least one of the animated facial mesh or the target video to the source video and generating a coarse output video that includes the transferred tracked expression. The method further includes refining the coarse video to generate a finished video, at least in part, by using a parameterized autoencoder trained on lip shapes in the animated facial mesh or the target video. One or more computers may perform the operations of the method.

[0007] The methods described herein provide an automated process for modifying film images or other video so that the lip shapes and timing of mouth movements of people appearing in the images appear more natural when accompanied by an audio recording that does not match the speech of the original video recording. Applications of these methods may include, for example, altering dialogue in video content during post-production, providing translated dialogue, or editing content after release to avoid criminalizing due to altered cultural norms or other cultural diversity factors, to avoid legal liability, to enhance creative impact, or for any other reason.

[0008] An apparatus for performing the method may include at least one computer processor coupled to a memory and one or more ports, the one or more ports including at least one input port and at least one output port (e.g., a desktop computer, a laptop computer, a tablet computer, a smartphone, a PDA, etc.). The computer processor may include, for example, a microprocessor, a microcontroller, a system on a chip, or other processing circuitry. As used herein, "processor" refers to a computer processor.

[0009] To the accomplishment of the foregoing and related ends, one or more examples include the features hereinafter fully described and particularly pointed out in the claims. The following description and drawings set forth certain illustrative aspects in detail and are indicative of but a few of the various ways in which the principles of the examples may be employed. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings and disclosed examples, all of which aspects and their equivalents being included herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features, nature and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings, wherein like reference numerals identify like elements accordingly throughout the specification and drawings.

[0011] Figure 1 is a schematic diagram illustrating an overview of matching lip movements and motions in digital video to alternative audio.

[0012] Figure 2 is a block diagram illustrating an example of a computer network in which the novel methods and apparatus of the present application may be used.

[0013] Figure 3 is a schematic diagram illustrating a computer used to perform the operations of the present method.

[0014] Figure 4 is a block diagram illustrating a concise overview of a method for matching lip movements and motions in digital video to substitute audio after the initial capture of image and sound.

[0015] Figures 5 to 8 It is an icon Figure 4 A flow chart illustrating more detailed aspects of the method is shown.

[0016] Figure 9 is an illustration using alternative wording Figure 4 Flowchart of the method and operation.

[0017] Figure 10 This diagram is used for Figure 9 A conceptual block diagram of components of an apparatus or system illustrating a method. DETAILED DESCRIPTION

[0018] Various aspects will now be described with reference to the accompanying drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will be apparent that various aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate description of novel combinations of these aspects and elements.

[0019] Figure 1An overview of matching lip sync and movement in a digital video with alternative audio 100 in the digital video is shown by one or more computer processors. As is known in the art, movement is animated in a video by a sequence of images that may be photographic, computer generated, hand drawn, or any combination of the foregoing. At 102, an original or "input" scene is recorded by an audio-video recording system in a digital audio-video. The recorded scene includes images of actors speaking original lines or improvisations (e.g., as dialogue or monologues) recorded in an audio track. Optionally, if the source audio-video is in an analog (e.g., film) format, the analog recording may be converted to digital audio-video at 104 by an analog-to-digital converter. In fact, one use case for the current technology includes dubbing new dubs and dialogue into old movies produced before the digital age. This technology is also useful for updating original digital content. At 106, an audio recorder records a new and different voice 106 that the producer wishes to replace the original audio with that used in the scene. The following three processes 108, 110, 112 will be described in detail. Figures 4 to 9 This technique is described in more detail in

[15] and corresponds to the three basic operations of matching the shape and movement of the actor's mouth to the replacement voice. This technique includes matching the shape and timing of the mouth movements to the spoken words and is not limited to matching the timing of the movements.

[0020] At a high level, a first computer process 108 creates a sequence of facial poses that includes lip shapes that are roughly synchronized when animating the new voice; in other words, the animated skeletal animation (armature) of the actor's mouth matches the desired alternative audio track. In some embodiments, the lip shapes can be sparse and limited to key points extracted from the video frames and then normalized, where a processor determines the key point geometry to match the new audio. A second computer process 110 creates a rough approximation of a matching digitally rendered audio-video file based on the pose sequence and three sets of neural network processing for mouth movements, facial expressions, and mesh poses. A third computer process 112 uses another machine learning process based on the digitally rendered video of a model speaking the alternative voice or its underlying animated mesh to refine the modified frames. The process sequence 108-110-112 outputs an audio-video of the scene 114 that can be stored in a production database for editing and compilation into video content for distribution via any suitable channel.

[0021] Figure 2A computer network 200 is shown in which the novel methods and apparatus of the present application may be used. One or more digital signal processors 212 (e.g., a server farm or cloud) interconnected via a local area network, wide area network 216, or other network can execute the processes and algorithms described herein to produce altered digital audio-video content that can be stored and distributed from a content server 220. In some embodiments, analog (e.g., film) content 224 is converted to digital form using any suitable conversion process (e.g., digital scanning). Content for processing or delivery to consumers can be pulled or pushed from the content server 220 to various clients via the WAN 216 and one or more wireless networks 218 (e.g., cellular phone and data networks) and one or more routers / modems / hotspots 208. Clients for playing the altered audio-video content may include, for example, smartphones 206, personal computers 204, notepad devices 202, projectors 210, and wearable virtual reality or augmented reality devices (not shown). The altered audio-video content can be transcoded into a suitable format for the target device before delivery.

[0022] See also Figure 3 , the digital signal processor 300 may include a CPU 302 and a random access memory (RAM) 314 that holds program instructions and data executed and processed by the processor 300. When the apparatus 300 is powered off or in an inactive state, the program instructions and data may be stored in a long-term memory, such as a non-volatile magnetic, optical, or electronic memory storage device 316. One or both of the RAM 314 or the storage device 316 may include a non-transitory computer-readable medium that holds program instructions that, when executed by the processor 302, cause the apparatus 300 to perform methods or operations as described herein. The program instructions may be written in any suitable high-level language, such as C, C++, C, or Java. TM , and compiled to generate machine language code for execution by the processor. Program instructions can be grouped into functional modules to improve coding efficiency and comprehensibility. It should be understood that even if these modules are identifiable as partitions or groups in the source code, they are not necessarily distinguishable as separate code blocks in machine-level coding. A code bundle pointing to a specific function can be considered to include a module, regardless of whether the machine code on the bundle can be executed independently of other machine codes. Other components of the device 300 may include, for example, a display output 330, an audio converter 320, an input / output port 308 for connecting to a network, and a user interface device 304 for receiving user input to guide the start of the process described herein.

[0023] Figure 4A useful three-step automated method 400 is illustrated for matching lip shapes and movements in digital video to substitute audio after video and audio capture. At 410, a processor generates an animated skeletal animation-like data structure, also referred to as a shape sequence or sparse geometry sequence, for the speaker's mouth 415 in the source video. Strictly speaking, sequence 415 is not a skeletal animation used in the animation process, but rather an equivalent data structure that can be mapped to a skeletal animation. The shape sequence can be optimized for use in the following process 420 for generating a rough video 425. At 420, the processor performs a complex, multi-step automated process that tracks the facial animation and mouth shape and movement inferred from the substitute audio and transfers the shape and movement to the source video. In some embodiments, process 420 generates an animated facial mesh with the emotional expressions inferred from the substitute audio and renders the animated mesh into a reference video. The reference video can also be referred to as a target video. In other embodiments, the target video can be created by video recording an actor speaking new lines while the substitute audio is being generated. Then, at 430, the processor refines the appearance of the mouth in the rough video 425 using a machine learning autoencoder trained on images of mouth movements in the reference video. The autoencoder process 430 refines the shape and appearance of the mouth to more closely match the reference video used for the replacement audio. The resulting final audio-video clip 435 reduces the mismatch (error) to a level below that which is perceptible to most or all audience members, thereby achieving the goals discussed above. Figures 5 to 8 Further details of processes 410, 420, and 430 are described.

[0024] Figure 5 Illustrated is a process 500 (410) for generating a sequence of facial gestures including mouth shapes from desired alternative audio for use in downstream video processing. The process 500 is based on the work of Karras et al. in “ Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion Some elements of image processing techniques are explained more fully in ACM Transactions on Graphics, Vol. 36, No. 4, Article 94 (July 2017). Suwajanakorn et al. Synthesizing Obama:Learning Lip Sync from Audio ACM Transactions on Graphics, Vol. 36, No. 4, Article 95 (July 2017) and Kumar et al. ObamaNet:Photo-realistic lip-sync from text ”31 stAlternative methods for generating lip shapes from text or recorded audio were disclosed in the 31st Conference on Neural Information Processing Systems (NIPS 2017). The technique described by Karras et al. may be most useful for process 500, but other techniques are also possible alternatives. The methods of Suwajanakorn et al. and Kumar et al. do not address modification of facial expressions to match the alternative audio and, therefore, are not as comprehensive as the technique described by Karras et al., but may still be useful depending on the application.

[0025] The process 500 operates by sampling the surrogate audio 502, dividing it into a continuous sequence of continuous audio windows, each of which is less than 1 second, for example 0.5 seconds. At 506, the processor normalizes the volume of each audio window so that the surrogate audio utilizes the full dynamic range of the selected audio format (e.g., 16kHz mono). The audio input is modeled as a combination of a linear filter (channel) and an excitation signal (vocal cords). At 508, the processor separates the resonant frequencies (format) into linear resonances 514, which carry phoneme information and nonlinear features 510 that can be mapped to emotional states via linear predictive coding (LPC). Other useful separation processes may include, for example, Mel-frequency cepstral coefficients (MFCC) and perceptual linear prediction (PLP).

[0026] The processor applies a deep neural network 504 to derive a time-varying sequence 518 of speech features (e.g., phoneme morphology). The processor parameterizes 512 the nonlinear features by any useful model for representing the actor's emotional state. To infer emotion and facial gestures from audio data, Karras et al. teach a data-driven machine learning process that produces an n-dimensional emotion vector 516 representing the actor's emotional state, where "n" is large enough to represent the emotional state that affects facial expression during speech, but not so large as to render the training process too specialized for the input. Manual encoding of emotional states can provide another option, or emotion vectors can be generated from other data (e.g., non-photographic biometric data collected during the recording of the reference video). If used, any useful method can be used to derive emotion data 516 associated with the actor's time speaking the lines in the reference video. The addition of the emotion state vector 516 and the input to the articulation map 520 enables the convincing use of alternative audio that conveys emotions different from the original performance.

[0027] The processor inputs the speech features 518 and the emotion vector 516 into a deep neural network process 520 for facial articulation, which outputs an abstract digital representation of the facial pose at the center (midpoint) of the audio window for each sample. This abstract representation is optimized for use in a third neural network process 524, which generates a geometric mesh 526 of vertices for three-dimensional (3D) rendering. In a more efficient alternative, since the pose sequence 526 cannot yet be used as the final output, the processor can generate other shape information optimized for use in downstream neural network processing (e.g., a set of 2D vertices representing a two-dimensional projection or other mapping of the mouth pose). The processor saves the pose mesh 524 in a format selected for the sample window in a computer memory that has sequence information for building a renderable animation model. At 530, the processor checks the condition for continuing the loop; if the desired pose sequence is not complete, the processor selects the next audio sample 532 and repeats the loop just described.

[0028] Once the sequence is complete, the processor generates and outputs 532 a sequence of shapes 415 in continuous order for use in downstream processing.

[0029] refer to Figure 6, the processor uses the shape sequence 415 in process 600 (420) to track the facial animation and mouth shape and movement in the reference video of the actor speaking the replacement line, and transfers the shape and movement from the reference video to the source video. Process 600 may use certain technical features described in "Face2Face: Real-time Face Capture and Reenactment of RGB Videos" by Nieβner et al. (downloaded from https: / / web.stanford.edu / ~zollhoef / papers / CVPR2016_Face2Face / paper.pdf, Stanford University 2016). Nieβner et al. describe a process in which a producer records or obtains a video of an actor delivering the lines that make up the alternative video and edits the video to create a target video 604 that is associated with the original video 606 to be modified. The processor provides the two input videos 604, 606 with the shape sequence 415 to a parameterization operation 610 that generates a parameterized facial model 612. Nieβner et al. describe a 3D model generated by principal component analysis (PCA) with 53,000 vertices and 102,000 faces. The exact number of vertices and faces is not important. The PCA process described by Nieβner et al. can be adjusted based on the size and resolution of the video being processed. For example, higher-resolution videos may require a higher-resolution model. In process 600, shape sequence 415 replaces or augments the facial tracking described by Nieβner et al.

[0030] In contrast to Nieβner et al., the current technique does not require a video of an actor speaking alternate lines. Instead, a reference video 604 can be rendered 602 from an animated sequence of facial poses (shape sequence 415) generated using process 500. Alternatively, the process of Nieβner et al. can be modified to use shape sequence 415 and associated metadata for parameterization 610, without requiring an intermediate rendering step 602 to produce reference video 604. For example, a parameterized facial model can be derived from shape sequence 415 rather than from RGB images captured from the target video data. Either way, generating a sequence of facial poses (rather than hiring and recording actors) can enable more cost-effective processing of traditional video content (both analog and digital), including feature-length film content.

[0031] Once the facial model is parameterized, the processor performs several operations to regularize any differences between the target video 604 and the original video 606. At 614, the processor adjusts the image parameters of the facial model to match the original video. At 616, the processor aligns the frames of the rendered facial model so that the rendered face is in the closest possible alignment with the face of the speaking actor in the original video. At 618, the processor statistically regularizes the facial model to promote plausibility of the synthesized face appearing in the output rough video, based on the assumption of a normally distributed population, by setting parameters to synthesize faces close to the population mean. At 620, the processor uses a data-parallel GPU to perform unconstrained nonlinear optimization of the facial model based on an iteratively reweighted least squares (IRLS) solver or other suitable optimization method. At 622, the processor bundles keyframes to solve the optimization algorithm on sets of adjacent keyframes. At 624, the processor transfers the expressions from the target video to the facial model. The processor may use subspace warping transfer techniques such as those taught by Nieβner et al. At 626, the processor synthesizes a realistic mouth region based on the mouth as it appears in the target video by finding the best matching frame for the mouth. A detailed description is provided by Nieβner et al. At 628, the processor renders a facial model for each frame and stitches the rendered facial model into its master frame. The rendered and stitched frames are saved as the rough output video 425.

[0032] After the rough video 425 is created, the automatic refinement process 700 is ready for execution by a digital signal processor, taking as input the rough video 425 of the actor speaking the replacement lines and the target video 604, the replacement lines having similar poses and image parameters as the original video. The processor can use automatic features to automatically crop the two videos 425, 604 to exclude portions of each frame except the mouth, lips, and adjacent portions of the cheeks, chin, and upper lip. The processor introduces random or systematic errors 712 into a training set 716 of images created from the target video 604 and uses the training set 716 to train a machine learning (e.g., neural network) autoencoder 714. Once training is complete 718, the processor passes the trained autoencoder 830 ( Figure 8) processes 720 the cropped input set 706. This process causes the autoencoder to modify the input set 710 as if the differences between it and the training set 716 were errors. The resulting corrected input set thus conforms to the training set 716 and, by logical extension, to the target video 704. The processor then generates a finished audio-video clip 724 by aligning the processed input set and inserting 722 the processed input set into the rough video 702 as a replacement for the originally extracted input set. Various alignment methods are known in the art, and because the processed input set is derived from the image into which it is inserted, the alignment error will be small or zero. The finished video 724 closely matches the modified speech, and all of the processing is automatic and feasible without requiring additional overhead.

[0033] Figure 8 The diagram illustrates an autoencoder training process 800 (714) suitable for generating a trained autoencoder 830 for use in method 700. Autoencoders include algorithms that perform compression and decompression functions on the same content, and these algorithms are 1) data-specific, 2) lossy, and 3) automatically learned from examples rather than predefined. As used herein, an "autoencoder" simply includes a module in which a machine learning algorithm (e.g., a neural network) performs the compression and decompression functions. As of May 2018, the open source program "Deepfakes" (https: / / github.com / deepfakes) is an example of an autoencoder that may be suitable for use in process 800.

[0034] Data-specificity means that an autoencoder can only process data similar to the one on which its machine learning algorithm has been trained. In contrast, many signal processing algorithms (e.g., MP3, MPEG-4, H.264, H.265, etc.) use predefined sets of rules that are data-independent, in addition to requiring the data to conform to the specifications defined for their format. For example, an autoencoder trained on images of faces will produce confusion on images of cars because it is parameterized through its training to process features specific to faces. Lossy refers to image degradation—a form of change that occurs during the autoencoding process. The lossy property enables the autoencoder to introduce changes to compensate for different inputs. Autoencoders can be trained to reduce the degradation to below a level that is perceptible to humans.

[0035] The autoencoder includes an encoding function 812 and a decoding function 816, which are characterized by a distance function (not shown) that defines the information loss between the compressed input data and the decompressed output of the autoencoder. The distance function may also be referred to as a "loss" or "error" function. The loss function may include, for example, the mean squared error between the input and output pixels of an input image. The encoder 812 and decoder 816 are parameterized functions, such as neural networks that are differentiable with respect to their distance functions. Therefore, the autoencoder can and should be parameterized using any suitable error minimization method (e.g., stochastic gradient descent) to minimize the processing loss.

[0036] The automatic encoding process 800 can be operated as follows. Before use, the automatic encoder is parameterized (meaning its parameters are configured) by machine learning based on a training set 802, which can include a set of images for each frame or selected frames of the target video 604. Figure 7 As described, the input image can be cropped to contain only the speaker's mouth. The automatic process can generate one or more modified images 806, 808, 810 from each target image 804, thereby introducing random and / or systematic distortions that can approximate similar differences in form and scale between the mouth shapes and movements in the rough video 425 and the target video 804. The automatic encoding process 800 processes each modified input image and adjusts its parameters to minimize the error between the target image and the corresponding modified input image. The parameter encoder 812 compresses the modified images (e.g., input 1 to N 806, 808, 810 in sequence) to create a compressed image 814 for each input image. The parameter decoder 816 decompresses the compressed image 814 to create an output image 818.

[0037] The error measurement and minimization process 820 measures the distance (error or loss) between the target image 804 and the output image 818. Unless and until the error falls below a threshold, error feedback in the form of parameter adjustments is provided to the encoding 812 and decoding 816 processes, and the input image is reprocessed. After the error is minimized, the process 800 can pick up the next image in its training set 824 and repeat the process just described. Similar repetitions occur until all images in the training set have been processed 826. The parameters of the autoencoder are then saved to an appropriate file format 830, and the autoencoder 830 is ready for use in conjunction with the above. Figure 7 The process 700 is described.

[0038] In accordance with the foregoing, and as an additional example, Figure 9The more general aspects of one or more methods 900 according to one embodiment are shown, as can be performed by a computer suitable for machine learning and digital signal processing (including digital audio and video). It should be understood that the more general operations of method 900 can include or embody more detailed aspects of the corresponding methods described above.

[0039] refer to Figure 9 , a computer-implemented method 900 for matching lip shapes and movements in a digital video to alternative audio after video and audio capture may include: at 910, deriving, by at least one computer processor, a sequence of facial poses including the lip shapes of an actor from a source digital video in which the actor appears, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample. As used herein, a "middle position" includes a midpoint and adjacent points that are biased toward the midpoint. While the midpoint is considered optimal, nearby positions may also provide useful results. The method may further include: at 920, generating, by at least one processor, an animated facial mesh based on the sequence of facial poses and the source digital video. In some embodiments, the method may include generating a target video from the animated facial mesh, the animated facial mesh depicting facial expressions of the mesh in a manner that is roughly aligned with and corresponds to a photograph of the source video. The method may further include: at 930, transferring, by at least one processor, the tracked expressions from at least one of the animated facial mesh or the target video to the source video, and generating a rough output video including the transfer of the tracked expressions. In combination with the above Figure 6 A more detailed description of operations 920 and 930 is shown and described. The method may further include generating a finished video by refining, by at least one processor, the coarse video at least in part by using a parameterized autoencoder trained on lip shapes in the target video.

[0040] At least one processor may perform additional operations, such as encoding the completed video and audio in a non-transitory computer-readable medium for storage and distribution, or perform additional operations in conjunction with Figures 4 to 8 or as subsequently described below. Not every one of these additional operations is necessarily performed in every embodiment of the method, and the presence of any one operation does not necessarily require that any other of these additional operations also be performed. All operations are performed automatically by a computer processor under the control of one or more algorithms or programs encoded in computer memory.

[0041] In one aspect, generating the sequence of facial gestures may further include sampling a sequence of audio samples obtained from a recording of a spoken conversation. In some embodiments, generating the sequence of facial gestures may include converting text to speech using a text-to-speech synthesizer.

[0042] Additionally, generating the facial pose sequence may further include extracting facial pose keypoints from the image of the actor, normalizing the keypoints, and applying principal component analysis (PCA) to the normalized keypoints to derive the facial pose. Generating the facial pose sequence may further include deriving mel-frequency cepstral coefficients (MFCCs) for each sample and mapping each MFCC coefficient to one of the mouth shapes using a recurrent neural network.

[0043] Combined with the above Figure 5 In other described embodiments, generating a sequence of lip shapes may further include separating linear features from nonlinear features, processing the linear features using a deep neural network for format analysis to generate a time-varying sequence of speech features, and processing the nonlinear features of facial joints using a deep neural network to generate facial poses at intermediate positions. In related aspects, generating a sequence of facial poses including lip shapes may further include generating a vertex-defined facial pose for output via a deep neural network, and storing the pose in a sequence of facial poses.

[0044] In other embodiments, Figure 6 As shown and described, transferring the tracked expression from the target video to the source video can further synthesize the mouth region and render each frame of the rough output video.

[0045] On the other hand, if combined Figure 7 As shown and described, refining the coarse video using a parametric autoencoder trained on lip shapes in a target video can further include generating a training set for the autoencoder by randomly altering frames in the target set. Refining the coarse video using a parametric autoencoder trained on lip shapes in the target video can further include cropping the coarse output video and corresponding regions of the target video around the actor's mouth. In a related aspect, refining the coarse video using a parametric autoencoder trained on lip shapes in the target video can further include aligning and inserting processed images from the target video into the coarse output video.

[0046] Figure 10 1 is a conceptual block diagram illustrating components of an apparatus or system 1000 for synchronizing modified speech and mouth movements following audio-video capture as described herein, according to one embodiment. As depicted, apparatus or system 1000 may include functional blocks that may represent functions implemented by a processor, software, or a combination thereof (e.g., firmware).

[0047] like Figure 10As shown, the apparatus or system 1000 may include an electrical component 1002 for deriving a sequence of facial poses including the actor's mouth shapes from a source digital video in which the actor appears, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample. Component 1002 may be or may include a device for said deriving. The device may include a processor 1010 coupled to a memory 1016 and an input device 1014, the processor executing an algorithm based on program instructions stored in the memory. For example, as described above in conjunction with Figure 5 Such an algorithm may comprise a more detailed sequence of operations as described.

[0048] The apparatus or system 1000 may further include an electrical component 1003 for generating an animated facial mesh based on a sequence of facial poses and a source digital video. The component 1003 may be or may include a device for said mesh generation. The device may include a processor 1010 coupled to a memory 1016 and an input device 1014, the processor executing an algorithm based on program instructions stored in the memory. For example, in combination with Figure 6 As described in block 602 of FIG. , this algorithm may include a series of more detailed operations. Figure 6 As described at 612 , the processor may render the target video before or after parameterizing the facial model.

[0049] The apparatus or system 1000 may further include an electrical component 1004 for transferring the tracked expression from at least one of the animated facial mesh or the target video to the source video based in part on the geometric facial model, and generating a rough output video including the transfer of the tracked expression. The component 1004 may be or may include a device for said transferring and generating. The device may include a processor 1010 coupled to a memory 1016 and an input device 1014, the processor executing an algorithm based on program instructions stored in the memory. For example, as in conjunction with Figure 6 Such an algorithm may include a more detailed sequence of operations as shown and described in blocks 624 through 628.

[0050] The apparatus or system 1000 may further include an electrical component 1006 for refining the rough video at least in part by using a parameterized autoencoder trained on the mouth shapes in the animated facial mesh or target video to generate a finished video. The component 1006 may be or may include a device for said generation. The device may include a processor 1010 coupled to a memory 1016 and an input device 1014, the processor executing an algorithm based on program instructions stored in the memory. For example, as in conjunction with Figure 7 and Figure 8 Such an algorithm may comprise a more detailed sequence of operations as shown and described, wherein Figure 7shows the application of an autoencoder to refine the mouth portion of an altered video, and Figure 8 Aspects and operation of useful autoencoders are shown.

[0051] In the case where the device 1000 is configured as a digital signal processor, the device 1000 may optionally include a processor module 1010 having at least one processor. In this case, the processor 1010 may be in operable communication with the modules 1002-1006 via a bus 1012 or other communication coupling (e.g., a network). The processor 1010 may initiate and schedule functions performed by the electrical components 1002-1006.

[0052] In a related aspect, the apparatus 1000 may include a network interface module 1014 operable to communicate with a storage device via a computer network. In a further related aspect, the apparatus 1000 may optionally include a module for storing information, such as a memory device / module 1016. The computer-readable medium or memory module 1016 may be operably coupled to other components of the apparatus 1000 via the bus 1012 or the like. The memory module 1016 may be adapted to store computer-readable instructions and data to implement the processes and behaviors of the modules 1002-1006 and their subcomponents, or the processor 1010, or methods 400-800, or one or more additional operations described in conjunction with the method 900. The memory module 1016 may retain instructions for executing the functions associated with the modules 1002-1006. Although the modules 1002-1006 are illustrated as being external to the memory 1016, it should be understood that the modules 1002-1006 may reside within the memory 1016.

[0053] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether these functions are implemented as hardware or as software depends on the application and design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each application, but such implementation decisions should not be interpreted as resulting in departure from the scope of this disclosure.

[0054] As used in this application, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer or collaborative computer system. For example, both an application running on a server and the server can be a component. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers.

[0055] Various aspects will be presented in terms of systems that may include several components, modules, etc. It should be understood and appreciated that various systems may include additional components, modules, etc. and / or may not include all of the components, modules, etc. discussed in conjunction with the figures. Combinations of these methods may also be used. Various aspects disclosed herein may be performed on electrical devices including devices that utilize touch screen display technology and / or mouse and keyboard type interfaces. Examples of such devices include computers (desktop and mobile), smartphones, personal digital assistants (PDAs), and other wired and wireless electronic devices.

[0056] In addition, the various illustrative logical blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0057] The operational aspects disclosed herein may be embodied directly in hardware, in software modules executed by a processor, or in a combination of the two. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium. In an alternative embodiment, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative embodiment, the processor and storage medium may reside in a user terminal as discrete components.

[0058] Furthermore, standard programming and / or engineering techniques may be used to generate software, firmware, hardware, or any combination thereof to control a computer so that one or more versions may be implemented as a method, apparatus, or article of manufacture to implement the disclosed aspects. Non-transitory computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), Blu-Rays, etc.), and media such as optical disks (e.g., optical disks (CDs), digital versatile disks (DVDs), Blu-Rays, etc.). TM ...), smart cards, solid state devices (SSDs), and flash memory devices (e.g., cards, sticks). Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the disclosed aspects.

[0059] The preceding description of the disclosed aspects is provided to enable one skilled in the art to make or use the present disclosure. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0060] In view of the exemplary systems described above, methodologies that may be implemented according to the disclosed subject matter have been described with reference to several flow charts. Although, for purposes of simplicity of explanation, the methodologies are illustrated and described as a series of blocks, it should be understood and appreciated that the claimed subject matter is not limited by the order of these blocks, as some blocks may occur in a different order and / or concurrently with other blocks depicted and described herein. Furthermore, not all illustrated blocks are required to implement the methodologies described herein. Furthermore, it should be further appreciated that the methodologies disclosed herein can be stored on an article of manufacture to facilitate communication and transfer of these methodologies to a computer.

Claims

1. A method for matching lip movements and motions in a digital video with substitute audio, the method comprising: deriving a sequence of facial poses including mouth shapes of an actor from a source digital video, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample of the substitute audio, at least in part by sampling a sequence of audio samples obtained from a speech dialog recording, separating linear features from non-linear features, generating a time-varying sequence of speech features by processing the linear features using a deep neural network for format analysis, and generating facial poses at the middle positions by processing the non-linear features of facial joints using a deep neural network; generating an animated facial mesh based on the facial pose sequence and the source digital video; transferring a tracked expression from at least one of the animated facial mesh or a target video rendered therefrom to the source digital video and generating a coarse output video including the transfer of the tracked expression; as well as A finished video is generated at least in part by refining the rough output video using a parameterized autoencoder trained on the animated facial mesh or the mouth shapes in the target video.

2. The method of claim 1, wherein deriving the sequence of facial gestures comprises converting text to speech using a text-to-speech synthesizer.

3. The method of claim 2, wherein deriving the facial pose sequence further comprises extracting key points for the mouth shape from the image of the actor, normalizing the key points, and applying principal component analysis (PCA) to the normalized key points to derive the mouth shape.

4. The method of claim 3, wherein deriving the facial gesture sequence further comprises deriving mel frequency cepstral coefficients (MFCC) for each of the samples, and mapping each MFCC coefficient to one of the mouth shapes using a recurrent neural network.

5. The method of claim 1, wherein deriving the facial pose sequence further comprises generating vertex-defined facial poses for output via a deep neural network, and saving the poses in the facial pose sequence. 6 . The method of claim 1 , wherein transferring the tracked expression from the target video to the source digital video further comprises synthesizing a mouth region and rendering each frame of the rough output video.

7. The method of claim 1, wherein refining the coarse output video using a parameterized autoencoder trained on lip shapes in the target video further comprises generating a training set for the autoencoder by randomly altering frames in a target set.

8. The method of claim 1 , wherein refining the coarse output video using a parameterized autoencoder trained on lip shapes in the target video further comprises cropping corresponding regions of the coarse output video and the target video around the actor's mouth.

9. The method of claim 1, wherein refining the coarse output video using a parameterized autoencoder trained on lip shapes in the target video further comprises aligning and inserting processed images from the target video into the coarse output video.

10. The method of claim 1, further comprising rendering a target video from the animated facial mesh, wherein transferring the tracked expression comprises transferring the expression from the target video to the source digital video.

11. An apparatus for matching lip movements and motions in digital video with substitute audio, the apparatus comprising a processor coupled to a computer memory, the memory retaining program instructions that, when executed by the processor, cause the apparatus to: deriving a sequence of facial poses including mouth shapes of an actor from a source digital video, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample of the substitute audio, at least in part by sampling a sequence of audio samples obtained from a speech dialog recording, separating linear features from non-linear features, generating a time-varying sequence of speech features by processing the linear features using a deep neural network for format analysis, and generating facial poses at the middle positions by processing the non-linear features of facial joints using a deep neural network; generating an animated facial mesh based on the facial pose sequence and the source digital video; transferring a tracked expression from at least one of an animated facial mesh or a target video rendered therefrom to said source digital video and generating a coarse output video including the transfer of said tracked expression; as well as A finished video is generated at least in part by refining the rough output video using a parameterized autoencoder trained on the animated facial mesh or the mouth shapes in the target video.

12. The apparatus of claim 11, wherein the memory retains further instructions for deriving the sequence of facial gestures at least in part by converting text to speech using a text-to-speech synthesizer.

13. An apparatus according to claim 12, wherein the memory retains further instructions for the following operations: deriving the facial pose sequence at least in part by extracting key points for the mouth shape from the image of the actor, normalizing the key points, and applying principal component analysis (PCA) to the normalized key points to derive the mouth shape.

14. The apparatus of claim 13, wherein the memory retains further instructions for deriving the sequence of facial poses at least in part by deriving mel-frequency cepstral coefficients (MFCCs) for each of the samples and mapping each MFCC coefficient to one of the mouth shapes using a recurrent neural network.

15. The apparatus of claim 11, wherein the memory retains further instructions for deriving the sequence of facial poses at least in part by generating vertex-defined facial poses for output via a deep neural network and saving the poses in the sequence of facial poses.

16. The apparatus of claim 11, wherein the memory retains further instructions for transferring the tracked expression from the target video to the source digital video at least in part by synthesizing a mouth region and rendering each frame of the rough output video.

17. An apparatus according to claim 11, wherein the memory retains further instructions for the following operations: refining the rough output video at least in part by using a parameterized autoencoder trained on lip shapes in the target video in the following manner, namely, generating a training set for the autoencoder by randomly changing frames in a target set.

18. The apparatus of claim 11, wherein the memory retains further instructions for refining the coarse output video using a parameterized autoencoder trained on lip shapes in the target video, at least in part by cropping corresponding regions of the target video and the coarse output video around the actor's mouth.

19. The apparatus of claim 11, wherein the memory retains further instructions for refining the coarse output video using a parameterized autoencoder trained on lip shapes in the target video, at least in part by aligning and inserting processed images from the target video into the coarse output video.

20. The apparatus of claim 11, wherein the memory retains further instructions for rendering a target video from the animated facial mesh, wherein transferring the tracked expression comprises transferring the expression from the target video to the source digital video.

21. A non-transitory computer readable medium for matching lip movements and motions in digital video with substitute audio encoded using program instructions which, when executed by a computer, cause the computer to: deriving a sequence of facial poses including mouth shapes of an actor from a source digital video, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample of the substitute audio, at least in part by sampling a sequence of audio samples obtained from a speech dialog recording, separating linear features from non-linear features, generating a time-varying sequence of speech features by processing the linear features using a deep neural network for format analysis, and generating facial poses at the middle positions by processing the non-linear features of facial joints using a deep neural network; generating an animated facial mesh based on the facial pose sequence and the source digital video; transferring a tracked expression from at least one of an animated facial mesh or a target video rendered therefrom to said source digital video and generating a coarse output video including the transfer of said tracked expression; as well as A finished video is generated at least in part by refining the rough output video using a parameterized autoencoder trained on the animated facial mesh or the mouth shapes in the target video.

22. An apparatus for matching lip movements and motions in digital video with substitute audio, comprising: Means for deriving a sequence of facial poses including an actor's mouth shapes from a source digital video, wherein each pose in the sequence of facial poses corresponds to a middle position of each sample of the substitute audio, at least in part by sampling a sequence of audio samples obtained from a speech dialog recording, separating linear features from non-linear features, generating a time-varying sequence of speech features by processing the linear features using a deep neural network for format analysis, and generating facial poses at the middle positions by processing the non-linear features of facial joints using a deep neural network; means for generating an animated facial mesh based on the sequence of facial poses and the source digital video; means for transferring a tracked expression from at least one of an animated facial mesh or a target video rendered therefrom to said source digital video and generating a coarse output video including the transfer of said tracked expression; as well as Means for generating a finished video at least in part by refining the coarse output video using a parameterized autoencoder trained on the animated facial mesh or mouth shapes in the target video.

Images