System and method for improving lip dubbing

Abstract

In various embodiments herein, a computer-implemented system is described, the system comprising: a processor; a memory coupled to the processor and storing processor executable instructions that, when executed, configure the processor to: receive an initial video data object V comprising a plurality of image frame data objects I; receiving a time-synchronized set of view parameters corresponding to the target audio data object A '; and processing the initial video data object V and the time-synchronized set of visual position parameters using a machine learning network to generate an output video data object V ', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized set of visual position parameters.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 301,947 filed on January 21, 2022 and U.S. Provisional Patent Application No. 63 / 426,283 filed on November 17, 2022, which are incorporated herein by reference in their entireties. Technical Field

[0003] Embodiments of the present disclosure relate generally to the field of machine learning for visual effects, and more particularly, to systems and methods for improving manipulation of lip movements in a video or image, such as to match dubbed video footage in a target language. Background Art

[0004] The amount of content available on television is increasing rapidly. Foreign films are becoming more popular in English-speaking countries, and international streaming platforms are making it easier for non-English speakers to access English-language content.

[0005] In order to better appeal to audiences who speak a different language than the language of the film in question, it is desirable to translate the film script and then dub it. However, audio dubbing alone does not match the speaker's lip movements and can result in timing inconsistencies. It is therefore useful to manipulate the lip movements to match the dubbed film in any given language. However, manual manipulation is not practically feasible given the enormous effort required per frame. Summary of the invention

[0006] Therefore, there is a clear and growing need for automated systems and methods that, given a video V and audio A in a language L, can manipulate V to obtain a video V' based on audio A' in language L' such that the lips in video V' match the audio A'. However, this presents a challenging technical problem because videos are typically of high quality and high resolution, such as 4K or higher, and subtle mismatches or slight noise can produce noticeable artifacts that should ideally be corrected and removed.

[0007] In various embodiments herein, a computer-implemented method is described in which a specially configured machine learning model architecture is trained and deployed to receive input data objects V (data representation of an original video consisting of original video frame objects I), A (data representation of original audio). In some embodiments, a data object A' (data representation of target audio) is provided, while in other embodiments, the data object A' is generated (e.g., a machine translation step may occur to generate A').

[0008] The machine learning model architecture may include a generator network, which may include but is not limited to a generative adversarial network (such as StyleGAN), trained as a generator to track relevant image features and reassemble them into a "hallucinated" or machine-generated output image or image portion based on a set of input hyperparameters. The machine learning model architecture in the proposed method generates a generated image frame data object I', and I' may be the entire image or an image portion specifically related to the mouth area of ​​a character (e.g., a human or an animated character) on the screen.

[0009] The parameters provided to the machine learning model architecture for generating I' are modified relative to I so that the new parameter definition for the new viseme at the corresponding timestamp is used to generate I'. Thus, the machine learning model architecture is trained to computationally "redraw" the character's mouth region based on a new target set of timestamped visemes that can be extracted from A'. The generated image frame data object I' is then used to replace part or all of the image frame data object I in whole or in part when reassembling or generating the output video data object V'.

[0010] A viseme may represent a set of several speech sounds that appear the same, for example, when lip-reading. In video processing, a viseme may contain data representing the location of each of one or more regions (or pixels thereof) of a face (e.g., the mouth region and surrounding regions) when a set of several speech sounds that appear the same are spoken. For example, in English, the words "pet" and "bet" are different in pronunciation. However, when observed visually (without sound), the visual appearance of the mouth region of the person speaking the two words appears to be the same.

[0011] This approach can be used, for example, to computationally automate previous manual lip dubbing / redrawing exercises, and overcomes problems associated with previous approaches of lip dubbing where a replacement voice actor / actress speaking the target language must match syllables to the original lip movements (resulting in timing or script inconsistencies in the target language) or where the lip movements on screen do not correspond correctly to the audio in the target language (mouth movement but no speech, or no action but the character is speaking).

[0012] Further embodiments include using a system for animation post-processing where the output video is generated directly based on a time-synchronized set of visemes, rather than drawing the initial face or mouth in the original video, and for example, drawing the mouth or facial region directly as part of the rendering step. This reduces the effort required to prepare the initial video for input.

[0013] In yet another additional embodiment, the viseme data is provided to a machine learning framework comprising a machine learning network (e.g., a generative network) that generates video in the absence of original input video, and based on a set of instructions or storyboard data objects generates a fully "hallucinated" video with correct mouth shapes and mouth movements corresponding to a target audio track.

[0014] Two technical challenges were identified in adapting the machine learning data model architecture for the proposed use: (1) a computational method needs to be developed to disentangle expression (visepoint and lip shape) from other features such as pose, lighting, and overall texture; and (2) the system needs to be adapted to realistically synthesize missing visepoints (this is useful when the correct visepoint is not present in the input video, such as when the input video is very short).

[0015] Disentanglement is used to identify and define sets of parameters that control the operation of a machine learning network (e.g., a generative network). Identifying and defining sets of parameters is not an easy task because it is difficult to control specific output aspects of a machine learning model, many of which are learned iteratively in a "latent space" that is difficult to interpret or study. Disentanglement is a technical method that enables a machine learning architecture system to manipulate certain properties of an image (such as the characteristics of the generated image related to the mouth area, so that the mouth area can be manipulated to match a new sequence of visual elements) without changing other aspects. Disentanglement can be used as a first step in computationally learning how to control a machine learning model, which is then used to generate new images (e.g., new images with a modified mouth area). The disentanglement method can generate controllable parameters for a machine learning network, which can be a generative network such as StyleGAN.

[0016] For example, using the output of the disentanglement method, the machine learning architecture system can be configured to keep the pose, lighting conditions, or identity of the image intact and only modify the expression. To do this, the machine learning architecture system learns a separate code or representation for each aspect of the image, such as identity, expression (i.e., viseme), and pose, and the system then maps it to a single code that can generate an image with the same expression, identity, and pose as the input code.

[0017] When the latent space is generated or maintained by an autoencoder or through a mapper in a generative network such as StyleGAN, the "code" refers to a low-dimensional vector that can be fed to a generator / decoder network to generate an image at inference time. Such codes are learned and are low-level representations of high-dimensional data such as images.

[0018] The latent code may be represented by, for example, a vector in a multidimensional space, which may be a high-dimensional space. When the latent space is disentangled, directions in the latent space are associated with corresponding characteristics or aspects of the image, such that when the vector value moves along the direction, the corresponding characteristics or aspects of the image are changed or modified accordingly. Such directions in the latent space that may be represented by a vector (e.g., a latent code) may only change the age or expression or posture of a character within an image.

[0019] Therefore, embodiments of the system architecture described herein are configured to exploit the disentanglement properties of the latent space to find the direction or vector of an "expression" (e.g., mouth shape or lip shape) within an image. If the image is unchanged, moving along this direction can generate all possible expressions while maintaining other aspects (e.g., pose, lighting, identity).

[0020] So a single code or a single latent code is a single "point" in the latent space. This "point" represents a specific identity in a specific pose, with a specific expression in the image.

[0021] In some embodiments, the process performed by the system may include training a neural network that can learn gestures, identities, or expressions individually as codes, indices, or sets of 2D or 3D points. To ensure that each code has the correct purpose, the system can replace certain codes (e.g., gestures) for an image with codes for another image (another gesture), and certain codes (e.g., gestures) for an image with codes for another image (another gesture), and can ensure that all aspects except those corresponding to the modified code (i.e., gestures) remain intact through supervised paired training.

[0022] View synthesis may involve generating 3D models with good realism (e.g., within an acceptable range defined by various visual metrics) at different poses and viewpoints. To this end, a virtual camera may be placed on a sphere with a textured 3D model, which may be located at the origin. Several images may be presented at different poses at different viewpoints. These synthesized images may then be fed to a generative image-to-image conversion model to make them more realistic (e.g., to optimize representation realism or a particular metric that serves as a proxy for realism). For example, this generative model may be a mapper that receives a synthesized image and maps it to a real image distribution. Examples of such networks may include a properly trained cycleGAN or StyleGAN (such as pSp).

[0023] The methods described herein are implemented in practice using a computer system that maintains a trained generative adversarial model, which is physically implemented on a computer memory and storage device by refining and optimizing the interconnection weights representing different hidden features in a latent space through a training process (e.g., an iterative training process).

[0024] In some embodiments, the computer system may include a computer server or a distributed resource-based "cloud" computing system, and may, for example, be used with or as part of a video generation / translation / rendering process so that, for example, during post-processing, an input video (animated or real-life) may be processed to generate an output video with audio in various output languages, where the mouth region of the output video has been manipulated to match the target audio.

[0025] As described in further embodiments of this document, further details are presented in the case of providing a two-stage network in an example system in which the system is configured to generate an output video corresponding to a target set of audio instruction sets (e.g., for lip dubbing or replacement purposes).

[0026] An optional initial stage may be implemented whereby the example system in the example embodiment identifies segments in an input video V with input audio A and a target output video V' with output audio A' where lip shapes are represented by different visemes, and is configured to generate corresponding finely reconstructed frames corresponding to the visemes of the output audio A' only for each frame of the identified segment of the input video V. This is a useful method to reduce the overall computational burden without the need to generate image frames with identical visemes. In this initial stage, sounds are aligned to identify similar visemes between sound segments or silent areas (e.g., both are silent). For these segments, a bypass method can be used to simply copy frames of the input video instead of using the system to generate frames. In this optional stage, a pipeline approach may be used where the identified frames / audio segments that require lip dubbing are provided to a processing pipeline for generation.

[0027] The system may be a physical computing device such as a computer server, and the server may include a processor coupled to a computer memory, such as a server in a processing server farm having access to computing resources.

[0028] As part of the processing, the system can utilize a two-stage machine learning model topology, such as a voice-to-lips model (e.g., a machine learning model architecture) working with a lip-to-image model (e.g., a machine learning model architecture). The voice-to-lips model is a network that is configured to receive the geometry of the lips and animate it based on the sound. The lip-to-image model is a network that is configured to receive the personalized geometry of the lips (based on the audio) and each frame that needs to be dubbed. Each frame to be dubbed is first analyzed to extract the existing lip shape (which can be used for masking purposes by generating a mask in some instances), and then this area corresponding to the given lip shape (e.g., the mask area) is "filled" based on the personalized geometry of the lips. For example, the filling can include repair. The voice-to-lips and lip-to-image model networks can be pre-trained for many different people or identities ("IDs"), but in inference, if the video is long enough, they are fine-tuned for each speaker. The geometry of the face (e.g., lips, chin, and / or jaw) helps the network better predict the final reconstructed image (output dubbing). During testing, multiple variants were used to address issues related to unrealistic lips, lip position changes, chin stretching, etc. For example, to address the chin position issue, additional keypoints can be used for chin tracking, and for lip modeling, an improved keypoint extraction model can be applied to obtain a more accurate lip shape. Architecture changes can facilitate "lip sticking", etc. These issues can be further adjusted by using weightless equalization and other model changes.

[0029] Identity, or ID for short, is a synonym for person or role. When one or more machine learning networks are trained, pre-trained, or fine-tuned, the goal of the training, pre-training, or fine-tuning is to preserve or capture the speech texture and style of a specific person or identity.

[0030] The voice-to-lip machine learning model architecture is configured to encode a set of input audio associated with an individual into audio tokens, which are converted into a set of synthetic lip or chip key point data adjusted for the individual. The model is instantiated and trained for use during inference. The voice-to-lip machine learning model architecture is trained to predict the lip key points of the individual based on any provided video with image frames capturing the individual speaking. The synthetic lip or chip key point data set adjusted for the individual is determined based on deviations from a specific identity template. The identity template for an individual can be generated, for example, based on the average of all extracted key points of a reference video corresponding to the individual.

[0031] The lip-to-image machine learning model architecture is configured to receive a target frame and a keypoint code dataset learned from the lip or jaw geometry in the target frame and produce a coarse reconstruction frame, then receive appearance codes based on at least the synthetic lip or chip keypoint dataset, and produce a fine reconstruction frame using at least both the coarse reconstruction frame and the synthetic lip or chip keypoint dataset. The model is instantiated and trained for use during inference.

[0032] The lip-to-image machine learning model architecture may utilize at least a height-width loss function that measures the difference in lip openness between a ground truth value and at least one of the coarse reconstructed frame and the fine reconstructed frame.

[0033] The appearance code may be a personalized codebook learned for the individual and used to establish a set of coefficients estimated based on at least the keypoint code data set for scaling feature vectors.

[0034] The voice-to-lip machine learning model architecture and the lip-to-image machine learning model architecture can be pre-trained on multiple different human individual identities and can also be fine-tuned during inference time on the individuals.

[0035] The target frame for replacement may be a mask frame masked based on a maximum area that a jaw may cover in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the mask frame defining a repair region for generating at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0036] The methods described herein are useful in reducing the overall post-processing effort required to generate modified videos based on an input video source (e.g., where lips are replaced in certain segments or image frames). In particular, specific adjustments are proposed that use improved machine learning methods to help improve the accuracy of the generated frames or portions thereof, which can be personalized for one or more individuals.

[0037] Machine learning methods can be used to help the system avoid the “uncanny valley” problem. For commercial implementation, it is important to have enough accuracy to avoid people being suspicious when watching the generated videos.

[0038] The mechanism may be implemented in the form of a post-processing visual effects pipeline, for example, for television shows, movies, etc.

[0039] According to one aspect, a computer-implemented system for generating an output video for lip dubbing is provided, the system comprising: a processor; a memory coupled to the processor and storing processor-executable instructions, the processor-executable instructions, when executed, configuring the processor to: receive an initial video data object V comprising a plurality of image frame data objects I; receive a time-synchronized viseme parameter set corresponding to a target audio data object A'; and process the initial video data object V and the time-synchronized viseme parameter set using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized viseme parameter set.

[0040] In some embodiments, the time-synchronized viseme parameter set corresponding to the target audio data object A' is a parameter set identified using a plurality of encoder networks adapted to disentangle expressions e and gestures p from a plurality of image characteristics.

[0041] In some embodiments, the plurality of encoder networks are iteratively trained to disentangle expression parameters e, posture parameters p, and residual parameters r from the plurality of image frame data objects I.

[0042] In some embodiments, the processor is configured to extract an initial expression / viseme code set corresponding to a spectrogram segment in an initial audio data object A corresponding to an initial video data object V.

[0043] In some embodiments, the initial video data object V is used to populate a viseme library of available visemes based on the initial expression / viseme code set.

[0044] In some embodiments, the processor is configured to compare the visemes identified in the target audio data object A' with the viseme library of available visemes based on the initial expression / viseme code set to identify a subset of visemes that are not in the viseme library and need to be synthesized.

[0045] In some embodiments, the processor is configured to generate new expression / viseme codes corresponding to the subset of visemes not in the viseme library.

[0046] In some embodiments, the processor is configured to generate the output video data object V' by replacing at least one image frame data object of the plurality of image frame data objects I in its entirety with a replacement image frame data object I'.

[0047] In some embodiments, the processor is configured to generate the output video data object V' by replacing a portion of at least one image frame data object of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to a mouth region.

[0048] In some embodiments, the processor is configured to iteratively generate the output video data object V' for each character present in the initial video data object V.

[0049] In some embodiments, the machine learning network comprises a generative adversarial network.

[0050] According to another aspect, a computer-generated method for generating an output video is provided, the method comprising: receiving an initial video data object V comprising a plurality of image frame data objects I; receiving a time-synchronized viseme parameter set corresponding to a target audio data object A'; and processing the initial video data object V and the time-synchronized viseme parameter set using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized viseme parameter set.

[0051] In some embodiments, the time-synchronized viseme parameter set corresponding to the target audio data object A' is a parameter set identified using a plurality of encoder networks adapted to disentangle expressions e and gestures p from a plurality of image characteristics.

[0052] In some embodiments, the plurality of encoder networks are iteratively trained to disentangle expression parameters e, posture parameters p, and residual parameters r from the plurality of image frame data objects I.

[0053] In some embodiments, the method may include: extracting an initial expression / viseme code set corresponding to a spectrogram segment in an initial audio data object A corresponding to an initial video data object V.

[0054] In some embodiments, the initial video data object V is used to populate a viseme library of available visemes based on the initial expression / viseme code set.

[0055] In some embodiments, the method may include: comparing the visemes identified in the target audio data object A' with the viseme library of available visemes based on the initial expression / viseme code set to identify a viseme subset that is not in the viseme library, and generating new expression / viseme codes corresponding to the viseme subset that is not in the viseme library.

[0056] In some embodiments, the method may comprise generating the output video data object V' by replacing in its entirety at least one image frame data object of the plurality of image frame data objects I with a replacement image frame data object I'.

[0057] In some embodiments, the method may comprise generating the output video data object V' by replacing a portion of at least one image frame data object of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to a mouth region.

[0058] In some embodiments, the method may comprise iteratively generating the output video data object V' for each character present in the initial video data object V.

[0059] According to yet another aspect, a system for generating an output video corresponding to a target set of audio instruction sets is provided, the system comprising: a processor; a memory coupled to the processor and storing processor-executable instructions, the processor-executable instructions, when executed, configuring the processor to: instantiate a sound-to-lip machine learning model architecture, the sound-to-lip machine learning model architecture being configured to encode an input audio set associated with an individual into audio tokens, the audio tokens being converted into a synthetic lip or chip key point data set adjusted for the individual; and instantiate a lip-to-image machine learning model architecture, the lip-to-image machine learning model architecture being configured to: receive a target frame and a key point code data set learned from the lip or jaw geometry in the target frame and generate a coarse reconstructed frame; receive an appearance code based at least on the synthetic lip or chip key point data set; and generate a fine reconstructed frame using at least both the coarse reconstructed frame and the synthetic lip or chip key point data set.

[0060] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

[0061] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is fine-tuned during inference on the individual.

[0062] In some embodiments, the voice-to-lip machine learning model architecture is trained to predict lip landmarks of an individual based on any provided video with image frames capturing the individual speaking.

[0063] In some embodiments, the synthesized lip or chip keypoint data set adjusted for the individual is determined based on deviations from an identity template.

[0064] In some embodiments, the identity template of an individual is generated based on an average of all extracted keypoints of reference videos corresponding to the individual.

[0065] In some embodiments, the target frame is a mask frame that is masked based on a maximum area that a jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the mask frame defining a repair region used to generate at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0066] In some embodiments, the appearance code is a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients estimated based on at least the keypoint code data set for scaling feature vectors.

[0067] In some embodiments, the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip openness between a ground truth value and at least one of the coarsely reconstructed frame and the finely reconstructed frame.

[0068] In some embodiments, the processor is further configured to: identify segments in an input video V with input audio A and a target output video V' with output audio A', where lip shapes represent different visual positions, and generate, for only each frame of the identified segment of the input video V, a corresponding finely reconstructed frame corresponding to the visual position of the output audio A'.

[0069] According to yet another aspect, a computer-implemented method for generating an output video corresponding to a target set of audio instructions is provided, the method comprising: instantiating a voice-to-lip machine learning model architecture, the voice-to-lip machine learning model architecture being configured to encode an input audio set associated with an individual into audio tokens, the audio tokens being converted into a synthetic lip or chip key point data set adjusted for the individual; and instantiating a lip-to-image machine learning model architecture, the lip-to-image machine learning model architecture being configured to: receive a target frame and a key point code data set learned from the lip or jaw geometry in the target frame and produce a coarsely reconstructed frame; receive an appearance code based at least on the synthetic lip or chip key point data set; and produce a finely reconstructed frame using at least both the coarsely reconstructed frame and the synthetic lip or chip key point data set.

[0070] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

[0071] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is fine-tuned during inference on the individual.

[0072] In some embodiments, the voice-to-lip machine learning model architecture is trained to predict lip landmarks of an individual based on any provided video with image frames capturing the individual speaking.

[0073] In some embodiments, the synthesized lip or chip keypoint data set adjusted for the individual is determined based on deviations from an identity template.

[0074] In some embodiments, the identity template of an individual is generated based on an average of all extracted keypoints of reference videos corresponding to the individual.

[0075] In some embodiments, the target frame is a mask frame that is masked based on a maximum area that a jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the mask frame defining a repair region used to generate at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0076] In some embodiments, the appearance code is a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients estimated based on at least the keypoint code data set for scaling feature vectors.

[0077] In some embodiments, the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip openness between a ground truth value and at least one of the coarsely reconstructed frame and the finely reconstructed frame.

[0078] In some embodiments, the method further comprises: identifying segments in an input video V with input audio A and a target output video V' with output audio A', wherein lip shapes represent different visemes, and generating, for only each frame of the identified segment of the input video V, a corresponding finely reconstructed frame corresponding to the viseme of the output audio A'.

[0079] In some embodiments, a non-transitory computer-readable medium is provided that stores a set of machine-interpretable instructions that, when executed by a processor, cause the processor to perform method steps according to any one of the above methods.

[0080] The system may be implemented as a dedicated machine, such as a dedicated computing device that may be part of or operate as a computer server. For example, a rack-mounted device that may be used in a data center for a specific purpose: receiving input video over a message bus as part of a processing pipeline to create output video. The dedicated machine is used as part of a visual effects post-production computing method, for example, editing after the initial material is produced. Editing may include the integration of overlays or introduced computer graphics elements to replace portions of live action footage or animation, and such editing may be computationally intensive.

[0081] A special purpose machine may be instructed according to a set of machine-interpretable instructions that cause a processor to perform the steps of a computer-implemented method. The set of machine-interpretable instructions may be attached to a physical non-transitory computer-readable medium as an article of manufacture, such as a tangible physical storage medium (e.g., an optical disk, a solid-state drive, etc.) that may be provided to a computer server or computing device to be loaded or executed with various programs.

[0082] In the context of the disclosed method, the pipeline receives input for post-processing, which may include a video data object and a target audio data object. The system is configured to generate a new output video data object that effectively replaces certain regions (such as regions in the mouth region).

[0083] This paper proposes variations of the computational architecture. For example, in the example embodiment, a single U-net is utilized which has shown strong performance in experimental analysis.

[0084] Variants of the masking method are also proposed, for example, an improved mask that extends the masking region to the nose region instead of just under the nose, which is also found to show strong performance in experimental analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In the accompanying drawings, embodiments are shown by way of example. It is to be expressly understood that the description and drawings are for illustration purposes only and as an aid to understanding.

[0086] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0087] Figure 1 is a drawing illustrating an example lip dubbing system in accordance with some embodiments.

[0088] Figure 2 is an illustrative diagram of a process for decomposing audio into phonemes and retrieving associated visemes, according to some embodiments.

[0089] Figure 3is a block diagram of an unwrapping process in which an image is encoded into an unwrapped code having all image information according to some embodiments.

[0090] Figure 4 is a block diagram of a lip dubbing process in which expression codes (visemes) are extracted from audio and added to the codes of input frames to obtain output frames synchronized with audio segments, according to some embodiments.

[0091] Figure 5 is a block diagram illustrating a disentanglement network training process in which a loss is defined on latent codes and on images from a database with correct poses and expressions, according to some embodiments.

[0092] Figure 6 is an illustrative diagram of a method for data synthesis under different postures and expressions (visemes) according to some embodiments.

[0093] Figure 7 is a flowchart block diagram depicting the pre-processing of input video and audio.

[0094] Figure 8 is described as Figure 4 A flow chart block diagram of the performance of the Lip Dubber system is shown.

[0095] Fig. 9 is a schematic block diagram of a computing system suitable for use in video generation, according to some embodiments.

[0096] Fig.10 is a schematic block diagram of a computer system according to some embodiments.

[0097] Fig.11 is the first audio signal w a A visual representation of the spectrogram segment of the audio signal is compared with a spectrogram unit of a second audio signal.

[0098] Fig.12 is a block diagram of the process used to perform lip dubbing.

[0099] Fig.13A A machine learning topology diagram of an example voice-to-lip network is shown in accordance with some embodiments.

[0100] Fig. 13B An example sequence sampler is shown in accordance with some embodiments.

[0101] Fig.14 is a machine learning topology diagram illustrating a voice-to-lip model according to some embodiments, the model being configured to extract lip keypoints, audio, and identity templates from a reference video corresponding to an individual.

[0102] Fig.15A is an example lip-to-image network according to some embodiments.

[0103] Fig. 15B is an example lip-to-image network according to some embodiments.

[0104] Fig.16 is an illustrative diagram showing two images defining a repair area.

[0105] Fig.17 is an example flow chart illustrating face generation according to some embodiments.

[0106] Fig.18A and Fig.18B is an example method flow according to some embodiments. Fig.18A Extends to Fig.18B .

[0107] Fig.19 is an example block diagram of components of a system for performing lip dubbing according to some examples.

[0108] Fig. 20 An example computational process flow is shown that may be used as part of a processing pipeline in a commercially practical implementation. DETAILED DESCRIPTION

[0109] The amount of content available on television is increasing rapidly. Foreign films are becoming more popular in English-speaking countries, and international streaming platforms are making it easier for non-English speakers to access English-language content.

[0110] In order to better appeal to audiences who speak a different language than the language of the film in question, it is desirable to translate the film script and then dub it. However, audio dubbing alone does not match the speaker's lip movements and can result in timing inconsistencies. Therefore, it is necessary to manipulate the lip movements to match a dubbed film in any given language.

[0111] Therefore, there is a clear and growing need for systems and methods that, given a video V and audio A in a language L, can manipulate V to obtain V' based on audio A' in language L' so that the lips in V' match the audio A'. For example, the audio A can be in English and the audio A' can be in French.

[0112] This presents a challenging technical problem because videos are typically of high quality and high resolution, such as 4K or higher, and subtle mismatches or slight noise can produce noticeable artifacts that should ideally be corrected and removed. As described herein, a solution is proposed to provide a system specifically configured to generate an improved video data object V' having a modified region (e.g., a mouth region covering the lips and surrounding areas).

[0113] In a variant, the technical solution further comprises a viseme synthesis step for synthesizing visemes that can be used to generate V' but are not present in the original V (e.g., the original audio language does not have an actor making specific lip or mouth expressions for the target viseme); and an unwrapping step, which can be used to identify the control parameters required to send a time-coded input viseme set (e.g., corresponding to A', an audio track in the target language) to the generator to generate V'.

[0114] Figure 1 is a drawing illustrating an example lip dubbing system 100 in accordance with some embodiments.

[0115] The system 100 includes an input 102 having a video V with audio A in a language L and audio A' which is a translated audio A in the language L'. An output 104 is a video V' with audio A' in the language L' arranged in such a way that frames of the video V' match frames of their respective audio A'.

[0116] In some embodiments, in addition to audio A' in language L', video V also contains frames F and audio A in language L. System 100 will manipulate frames F such that each frame I∈F is manipulated to obtain I'∈F' that matches audio A'.

[0117] To match frames to audio segments, a deep neural network can be implemented that receives a frame I∈F and its corresponding spectrogram unit s∈A′ and produces a frame I′ that matches s.

[0118] Figure 2 is an illustrative diagram of a process 200 of decomposing audio into phonemes and retrieving associated visemes, according to some embodiments.

[0119] In phonology and linguistics, there are phonemes and visemes. A phoneme is a unit of sound that distinguishes one word from another in a particular language. For example, in most dialects of English, the sound pattern (sin) and (sing) are two different words that can be distinguished by replacing one phoneme / n / with another phoneme / η / .

[0120] A viseme is any of several sounds that look the same, for example, when lip-read. It should be noted that visemes do not enjoy a one-to-one correspondence with phonemes. For a particular audio track, phonemes and visemes can be time-coded as they appear on the screen or in the audio, and this process can be performed automatically or manually. Therefore, A' can be represented in the form of a data object that can contain a time-series encoded set of phonemes or visemes. For the phoneme representation, in some embodiments, it can be converted into a viseme representation (if available) by a lookup transformation. In another embodiment, the phoneme / viseme connection can be obtained by training a machine learning model through an iterative cycle with a supervised training data set with phonetic transcriptions and corresponding frames as value pairs.

[0121] Often, a single viseme can correspond to multiple phonemes, because several different phonemes appear the same way when produced on the face or lips. For example, words like pet, bell, and men are difficult for lip readers to distinguish because they all look like / pet / . Or phrases like "elephant juice" look exactly the same as "I love you" when lip-read.

[0122] As an example of a time series coded phoneme set, and also as Figure 2 As shown, for A', a time-stamped phoneme list marking the entire sequence can be generated based on the detected phonemes.

[0123] A time series of encoded visemes represents key points. For example, for each frame in the source video (depending on the frame rate of the source video), a set of key points is retrieved or generated, which indicates a new viseme that matches the frame. If there are 600 frames in the source video, there may be 600 key point sets. In this case, the time series or timestamps can contain frame correspondence.

[0124] Phoneme time coding (the set of phonemes used to produce a time-series code) can be viewed as operating on a "continuous" time space (although the audio is still sampled), whereas viseme time coding (the set of visemes used to produce a time-series code) is encoded into a discrete frame space.

[0125] A phoneme-to-viseme (P2V) codebook may be used to classify various different phonemes and their corresponding visemes. For example, a P2V codebook may be a data structure representing a lookup table that provides a classification of phonemes with corresponding visemes. The classification is not always 1:1, as many phonemes may have the same viseme, or similarly, contextual clues may change the viseme associated with a particular phoneme. By disentangling the viseme from other characteristics of the image, other characteristics of the face (e.g., anger) may be preserved.

[0126] Starting from an audio signal A and the associated text, decompose the audio A into segments s i To find the corresponding phoneme p i According to p i To determine or extract the corresponding viseme v i If the desired viewpoint and pose are available in the input video (see Fig.11 ), it can be retrieved from the original input, otherwise it may need to be generated as described herein using the proposed disentanglement model. In some embodiments, the desired viseme can also be obtained from a library associated with a specific actor or role in other speaking roles in other videos.

[0127] Visemes are added to a viseme database that can be pre-synthesized, as described further below.

[0128] Ideally, any lip movement could be constructed by combining images representing these visemes.

[0129] However, such images depicting a specific viseme are different in terms of pose, lighting conditions and other different factors. Therefore, a mechanism can be applied to manipulate these frames to match a specific viseme (expression), pose, lighting conditions and other different factors.

[0130] In some embodiments, the process includes classifying visemes or learning a code for each image. Then, by replacing one code and changing the others, the machine learning model architecture ideally only changes one aspect of the image (e.g., the relevant mouth area).

[0131] The code may be a vector of length N. Depending on the machine learning model architecture that may include a generator network such as StyleGAN, the length N and how the code is determined may be different.

[0132] For example, learning code examples such as Figure 3 The machine learning model architecture can learn the code by finding some code that produces the same image when fed to the generator network.

[0133] The machine learning model architecture is then trained to find the modifications needed to the code to generate the desired viseme while maintaining all other characteristics of the image. When modifying an image, the code for the shape of an “open mouth” of a person in the image should not turn their hair red.

[0134] As a non-limiting example of process 200, audio with text may be received and phonemes may be extracted from the received audio. Appropriate visemes may then be assigned to these recognized phonemes, which may be accomplished by looking up the corresponding visemes using a suitable P2V codebook.

[0135] Each frame I∈F consists of an expression e containing the geometry (ie, visemes) and texture of the lips and mouth, an identification string or number (ID) that distinguishes one individual from another, and a pose p that specifies the orientation of the face.

[0136] In the dubbing application, only the relevant facial expressions can be modified based on the spectrogram s, while the pose p and everything else (residue r) can remain as is. Therefore, the core neural network learns to disentangle e, r, and p from I.

[0137] Figure 3 is a block diagram of dewrapping 300 in which an image is encoded into a dewrapped code that preserves all image information, according to some embodiments.

[0138] Disentanglement is a technique that decomposes or disentangles features into narrow variables and encodes them into separate dimensions. The goal of disentanglement is to simulate the fast intuitive process of the human brain using both "high" dimensional reasoning and "low" dimensional reasoning.

[0139] In the example embodiment shown, disentanglement 300, image frames 310, 320 are processed by multiple encoders 330 into three disentangled codes, which represent pose, expression (viseme) and residue with all image information. For training, identities and pairs of images with the same pose, identity or ID should be retained. Paired data for disentanglement can be packaged or represented in different forms (e.g., vectors, integers, 2D / 3D points, etc.). In some embodiments, the method includes intentionally overfitting the input video to achieve improved results.

[0140] The described non-limiting neural network uses three encoders that are used to disentangle the expression e and pose p from other characteristics of the image, including other image characteristics such as ID, background, lighting, etc. The codes of these image characteristics are integrated into the code w through a multi-layer perceptron (MLP) network 340a, 340b. + 350a, 350b. + 350a, 350b can be passed to a pre-trained generator 360a, 360b such as StyleGAN to generate new images I' 370, 380.

[0141] The MLP network 340a, 340b is a type of neural network and is composed of one or more layers of neurons. Data is fed into an input layer, then there may be one or more hidden layers providing a level of abstraction, and then predictions are made at an output layer or "visible layer".

[0142] The encoder 330 and MLP networks 340a, 340b can be trained on an identity task, meaning that I and I' are identical, or on a paired dataset where I and I' are paired and differ in one or two features (e.g., ID, pose, or expression). For the purpose of lip dubbing, the expression can be obtained from a viseme database. During output video generation, I' can be a full image or a selected mouth region, and either can be inserted to generate a replacement video frame. Inserting only the mouth region may be faster and computationally cheaper, but this may have issues with bounding box regions, as well as inconsistencies with other aspects of the video that are not within the replacement region.

[0143] The training is described in further detail below.

[0144] Figure 4 is a block diagram of a lip dubbing process 400 in which expression codes (visemes) are extracted from audio and added to the codes of input frames to obtain output frames synchronized with audio segments, according to some embodiments.

[0145] Here the code for the input frame can be generated using a latent space inversion (or encoding) process.

[0146] Modifications to the vectors or codes allow the image to be semantically modified as it is passed back through the generator. For example, moving in the direction of “age” represented by the vector in the latent space will make the person in the generated image older.

[0147] An image frame I 410 is processed by multiple encoders 430a, 430b, 430c into three untangled codes representing pose p, expression (viseme) e, and residue r with all image information.

[0148] The non-limiting example process 400 herein implements three encoders 430a, 430b, 430c for disentangling the expression e and pose p from other image features, including ID, background, lighting, etc. The codes for these image features are integrated into the code w via a multi-layer perceptron (MLP) network 440. + 450 in. + 450 can be passed to a pre-trained generator 460 such as StyleGAN to generate a new image I'470.

[0149] In some embodiments, separate audio tracks are obtained for each individual character (or extracted from a combined audio track). For example, faces may be detected using a machine learning model to create normalized bounding boxes to identify heads and faces. Distal heads and proximal heads may have different methods because proximal heads may have a larger number of pixels or image areas to modify, while more distal heads may have a smaller number of pixels or image areas to modify.

[0150] To perform the lip dubbing process 400 shown in the example embodiment, expression codes (visemes) are extracted from the audio and added to the code of frame I to obtain frame I' which is synchronized with the audio segment.

[0151] To perform lip dubbing, the audio A' undergoes a viseme recognition process such that for each spectrogram segment s i Finding visemes. The system may be configured to map audio to phonemes, and then map phonemes to visemes.

[0152] For example, 19 visemes may be considered and indexed by a single unique integer (1-19). The spectrogram si may then be passed to another encoder or a separate module such as a phoneme-to-viseme module to generate the spectrogram from si. i e s The input video may or may not have the same viseme with the same pose as I. If V already has the same viseme and pose, it can be simply retrieved (see Fig.11 If it does not have, I is first encoded into three potential codes containing eS, r and p. Then, e s , r and p are passed to the decoder to generate a new frame I', which retains the ID, posture, etc., and is consistent with the expression e from the audio. s match.

[0153] In some embodiments, only the mouth region may be extracted from I' and inserted into I, and the images harmonized to generate a smooth result.

[0154] It should be noted that the latent codes can be of any size or form, including one-hot codes, single integer values, or floating point vectors of any size.

[0155] In other embodiments, it may be preferable to copy the entire I' or just create the lip shape and insert it back into I.

[0156] Additionally, if the correct pose and expression are already available in the input video V, suitable frames can simply be retrieved from the video V. In the absence of such frames, new frames can be generated using the discussed process. The example generator described may use StyleGAN or its variants.

[0157] In some embodiments, an additional feedback process is contemplated using a lip-reading engine that automatically generates output video / text which is then fed back into the system to be compared with the input to ensure that the output video is authentic.

[0158] Figure 5 is a block diagram of a disentanglement network training process 500 in which a loss is defined on a latent code and on images from a database with correct pose and expression, according to some embodiments.

[0159] For the training process 500 of what may be a first disentangled network, I and I' are already paired and have been improved in realism by pSp according to some embodiments.

[0160] Pixel2style2pixel (pSp) is an image-to-image translation framework. The pSp framework provides a fast and accurate solution for encoding real images into the latent space of a pre-trained StyleGAN generator. In addition, the pSp framework can be used to solve various image-to-image translation tasks, such as multimodal conditional image synthesis, face frontalization, inpainting, and super-resolution.

[0161] In some embodiments, pSp can be used to map images created in a synthetic environment with different viewpoints, poses, and textures to images that appear real.

[0162] To this end, the composite image can be fed to pSp and the code w0 can be generated.

[0163] In other embodiments, sampling may also be performed in a real domain called w1. By blending the top entries of w0 with the bottom entries of w1, the expression (e.g., viseme) and pose of the synthetic image captured in w0 may be preserved, and a real image may be produced that looks similar to the real image with code w1. By sampling different images and generating various w1s, some embodiments may generate a large number of labeled real images in certain poses and visemes indicated by the synthetic data. Such labeled real data may be used to learn disentanglement.

[0164] The loss L1 (e.g., |xi-xj|) can be defined based on the result and the ground truth. I' can also be fed back to the video encoder to obtain r', p', and e' and compare them with the input code. To do this, r', p', and e' can be defined as the function of i, p, and e s To ensure that the new lips are valid, the database should be searched for matches with r, p, and e. s The closest image.

[0165] Figure 6is an illustrative diagram of data synthesis 600 under different postures and expressions (visemes) according to some embodiments.

[0166] To understand the different characteristics of the wrapped frames, a relevant dataset is needed. To generate such a dataset, data can be synthesized with different identities rendered in different poses and expressions. These expressions contain all the available visemes required to produce effective lip dubbing.

[0167] The uncanny valley in aesthetics is a hypothesized relationship between the degree to which an object resembles a human and the emotional response to that object. The hypothesis states that humanoid objects that do not fully resemble real humans evoke the familiar sense of weirdness and disgust of "horror" in observers. The "valley" refers to a sharp drop in the affinity of a human observer to a replica that otherwise increases with the replica's human resemblance. For example, certain realistic robotic dolls that look almost human risk eliciting cold, weird feelings in viewers.

[0168] To overcome the uncanny valley and produce more realistic images, a collection of synthetic data will be fed to pSp to generate natural images with different IDs.

[0169] Thus, according to some embodiments, the described system learns to disentangle expression (vise position and lip shape) from other features such as pose, lighting, and overall texture. Hence, data is needed to learn how to disentangle these features.

[0170] Other embodiments realistically synthesize missing visemes. This is needed when the correct viseme is not available in the input video. This may be particularly useful when the input video is short. According to some embodiments, this is done by utilizing the system to generate synthetic data in different poses and IDs, and the additional steps described in data synthesis 600 above may be performed to make it more realistic.

[0171] Figure 7 is a flowchart diagram 700 depicting pre-processing of input video and audio.

[0172] In some embodiments, lip dubbing may consist of two parts. Flowchart 700 depicts the first part, pre-processing. In pre-processing, visemes of input 102 are found and added to a database. Audio A' is processed to identify viseme codes of its audio segments.

[0173] The second part according to the described embodiment relates to lip dubbing.

[0174] Figure 8 It depicts Figure 4 A flowchart diagram of the lip dubbing performance is shown in block diagram 800. Based on the viseme code of audio A', Figure 4The lip dubber depicted in can be used to modify frames of a video V.

[0175] Fig. 9 is a schematic block diagram of a computing system 900 suitable for use in video generation, according to some embodiments.

[0176] The system may be implemented by a computer processor or a set of distributed computing resources provided by a system for generating special effects or modifying video input. For example, the system may be a server specifically configured to generate a video output of lip dubbing, where an input video is received and a translation subroutine or process is executed to modify the input video to generate a new output video.

[0177] As described above, system 900 is a system based on a machine learning engine, the system comprising various maintained machine learning models that are iteratively updated and / or trained, with interconnected weights and filters that are adjusted to optimize for specific features (e.g., by a defined loss function). Multiple machine learning models can be used together in coordination, for example, as described herein, a specific set of machine learning models can be first used to untangle specific parameters, thereby ultimately controlling the video generator hallucination network.

[0178] Fig. 9 The computing elements shown are shown as examples and may vary, and more, different, or fewer elements may be provided. Furthermore, the computing elements may be implemented in the form of computing modules, engines, code routines, logic gate arrays, etc., and in some embodiments, the system 900 is a dedicated machine suitable for video generation (e.g., a rack-mounted device at a computing data center coupled to an input feed via a message bus).

[0179] This system can be used, for example, to computationally automate previous manual lip dubbing / redrawing exercises, and overcome problems associated with previous methods of lip dubbing, where a replacement voice actor / actress speaking the target language must match syllables to the original lip movements (resulting in timing or script inconsistencies in the target language) or where the lip movements on screen do not correspond correctly to the audio in the target language (mouth movement but no speech, or no action but the character is speaking).

[0180] An input data set is obtained at 902, for example, as a video feed provided by a studio or content creator, and may be provided, for example, as streaming video, as a video data object (e.g., .avi, .mp4, .mpeg). The video feed may have an associated audio track that may be provided separately or together. The audio track may be broken down by different audio sources (e.g., different feeds from different on-screen characters from a recording studio).

[0181] The target audio or script may be provided, but in some embodiments, it is not provided and the target audio or script may be synthesized using machine learning or other generative methods. For example, rather than having a new voice actor speak in a new language, the method obtains a machine translation and automatically uses the generated voice.

[0182] The viseme extraction engine 904 is adapted to identify the necessary visemes and their associated time codes from the target audio or script. In some instances, these visemes may be extracted from phonemes if provided, or from the video using a machine learning engine. Visemes may be mapped to a list of all visemes and stored as tuples (e.g., viseme 14, t=0.05-0.07 seconds, character Alice; viseme 13, t=0.04-0.08 seconds, character Bob).

[0183] The viseme synthesis engine 906 is configured to compare the necessary visemes with the known viseme set from the original video data object, and perform synthesis on the visemes missing from the original video data object as needed. This synthesis may include obtaining visemes from other works of the same actor, generating all new mouth movements based on the "eigenface", etc.

[0184] The viseme disentanglement engine 908 is a set of machine learning models that are individually tuned to decompose or separate movements related to mouth movements associated with various visemes when controlling a machine learning generator network 912, which is then used to generate control parameters using a control parameter generator engine 910.

[0185] The machine learning generator network 912 (e.g., StyleGAN or another network) is then operated to generate new frame objects whenever the person or character speaks or based on the viseme time code of the target viseme. The frame objects can be partial frames or full frames, and in some embodiments, are inserted into V to arrive at V'. In some embodiments, V' is not inserted into V, but is simply generated entirely by the machine learning generator network 912.

[0186] The output data set 914 is provided to a downstream computing mechanism for downstream processing, storage or display. For example, the system can be used to generate a translation of an ongoing event (e.g., a news broadcast), a movie / television program / animation output in multiple different languages ​​to a certain extent. In another embodiment, the output data set 914 is used to re-dub a character in the same language (e.g., in the case where the original audio is not available for some reason or simply does not meet the expectations). The system can also be used to modify accents (e.g., different English accents, Chinese accents, etc. can be corrected).

[0187] For example, the output data set 914 can be used for post-processing of animations, where instead of drawing the initial face or mouth in the original video, the output video is generated directly based on the time-synchronized viseme set, and the mouth or facial area is drawn directly as part of the rendering step, for example. This reduces the effort required to prepare the initial video for input.

[0188] In yet another additional example, viseme data and a system for generating video without original input video is provided, and based on a set of instructions or storyboard data objects, a completely "hallucinated" video is generated that corresponds to the correct mouth shape and mouth movement for a target audio track.

[0189] Fig.10 is an example computing system according to some embodiments. Under software control, computing device 1000 may be configured to Fig. 9 The schematic box shown is used to control the machine learning model architecture.

[0190] As shown, computing device 1000 includes one or more processors 1002, memory 1004, a network controller 1006, and one or more I / O interfaces 1008 that communicate via a message bus.

[0191] Processor 1002 may be one or more Intel x86, Intel x64, AMD x86-64, PowerPC, ARM processors, etc.

[0192] The memory 1004 may include a random access memory, a read-only memory, or a persistent storage device, such as a hard disk, a solid-state drive, etc. A read-only memory or a persistent storage device is a computer-readable medium. A computer-readable medium (e.g., a non-transitory computer-readable medium) may be organized using a file system and controlled and managed by an operating system that governs the overall operation of the computing device.

[0193] The network controller 1006 serves as a communication device to interconnect the computing device with one or more computer networks, such as a local area network (LAN) or the Internet.

[0194] One or more I / O interfaces 1008 may be used to interconnect the computing device with peripheral devices (e.g., keyboard, mouse, video display, etc.). Such peripheral devices may include a display of device 120. Optionally, network controller 1006 may be accessed through one or more I / O interfaces.

[0195] The software instructions are executed from a computer-readable medium by the processor 1002. For example, the software may be loaded from persistent storage of the memory 1004 or from one or more devices into random access memory through the I / O interface 1008 for execution by the one or more processors 1002. As another example, the software may be loaded and executed directly from a read-only memory by the one or more processors 1002.

[0196] Example software components and data stored within memory 1004 of computing device 1000 may include software that performs machine learning to generate hallucinated video data output, as disclosed herein, and operating system (OS) software that enables communications and application operations associated with computing device 1000 .

[0197] According to one embodiment of the present application, video V (image frame F+sound w a ) is given in language L (e.g., English), and the sound w b Given in language L' (eg, French). Figure 11-17 Various processes are shown that can be used to replace lip shapes in a video V according to sounds in a language L'.

[0198] Fig.11 is the first audio signal w a A visual representation 1100 of a comparison of a spectrogram segment of an audio signal with a spectrogram unit of a second audio signal. F 1102 shows a set of example time-stamped frames.

[0199] First Audio a The signal 1104 may be an audio signal of an audio A' in a language L'. b 1106 may be an audio signal of an audio A in a language L in the input video V. The second audio signal w b Each segment in the spectrogram segment of can have a known viseme and pose that can be obtained from the input video V. The audio signal w a Can be combined with audio signal b Align to identify the first audio signal w a The same second audio signal w b Spectrogram segment.

[0200] Audio signal a Can be combined with audio signal b Align to determine the second audio signal w b As shown in the depiction, the first audio signal w a Some spectrogram segments may be associated with the second audio signal w b Some of the spectrogram segments are identical ( Fig.11). For each common spectrogram segment, the known viseme and pose corresponding to the spectrogram segment of the first audio signal may be retrieved and used to determine the viseme and pose of the spectrogram unit of the second audio signal.

[0201] In some embodiments, frames of video V that match these common spectrogram units can be copied from video V and used to generate video V'. For the remaining spectrum segments that do not have commonality, Fig.12 The process shown.

[0202] This is an optional step that can be used to bypass certain similar frames to reduce overall computation time. For example, the sample output of this stage can be identified as segments that require frame generation (e.g., identified by time frame or duration). For example, these segments can represent all frames between two timestamps. For example, there may be a video in which there is speech between two people from t=5 seconds to t=6 seconds. However, it has been identified that similar frames exist for certain speech from t=5.00 seconds to t=5.3 seconds and from t=5.5 seconds to t=6.0 seconds. Therefore, the frames from t=5.3 seconds to t=5.5 seconds can be inserted from generation into the processing pipeline to generate frame portions representing replacement mouth portions of these frames. Each of the frames can be processed together using two trained networks to replace its mouth portion, as described below.

[0203] Fig.12 is a block diagram 1200 of a process for performing lip dubbing.

[0204] The process described can be used in cases where frames cannot simply be copied from the input video V, such as with respect to Fig.11 As explained. As depicted, the process can include a voice-to-lips step and a lips-to-image step. As shown in FIG. 18 , the described lip dubbing process can be performed using a system 1800. System 1800 can be a part of system 900 and can include a voice-to-lips network and a lips-to-image network. The voice-to-lips network can be a transformer neural network.

[0205] A transformer neural network is a neural network that learns context and therefore meaning by tracking relationships in sequence data. A voice-to-lip network can be used to personalize the geometry of the lips based on the speaker (through fine-tuning). The voice-to-lip step may involve receiving the geometry of the lips and animating the lips based on the sound or audio signal.

[0206] The lips to image step may involve receiving the personalized geometry of the lips (according to the audio) and each frame to be dubbed. As will be described in further detail below, each frame to be dubbed may first be analyzed to extract the existing lip shape for the purpose of masking the lips and chin.

[0207] As will be described in further detail below, the lips to image step can then be tasked with "filling in" this masked region corresponding to a given lip shape. Masking is a critical step because without it, the network cannot learn anything and can only copy from the input frame.

[0208] like Fig.12 As shown, there is a pre-training step 1202 and an inference step 1204 .

[0209] During the pre-training step 1202, both the voice-to-lips 1206 and lips-to-image 1208 models are trained, for example using individual identities or identity+shift pairs, so that the model interconnections and their weights are refined in a set of training iterations. Depending on what is available in the training set, a different set of faces can be trained.

[0210] During the inference step 1204, both the voice-to-lips 1206 and lips-to-image 1208 models may be fine-tuned for a particular individual before inference is made for that particular individual.

[0211] Voice to lips

[0212] Fig.13A An example voice-to-lip network 1300 is shown in accordance with some embodiments. The voice-to-lip network can use a transformer-based architecture. The voice-to-lip network can use a transformer-based architecture. The voice-to-lip network can be trained end-to-end to autoregressively synthesize lip (and chin) keypoints to match the input audio. As shown, the transformer model can include a transformer encoder (TransformerEncoder) that encodes the input audio into "tokens", and a transformer decoder (TransformerDecoder) that processes the audio tokens and previous lip keypoints to synthesize a sequence of lip keypoints. Transformer Encoder and Wav2Vec2.0 TM The design matches and can be initialized with its pre-trained weights. Wav2Vec2.0 is a self-supervised learning model for speech representation.

[0213] The Wav2Vec2.0 model is trained on 53,000 hours of audio (CC BY 4.0 licensed data), making it a powerful speech encoder. TMCompared to the Faceformer model, this application focuses on the explicit generation of lips (rather than full faces), and the personalization of lips for new identities not in the training set. TM is a transformer-based autoregressive model that encodes long-term audio context and autoregressively predicts a sequence of animated 3D face meshes.

[0214] The voice-to-lip model aims to address three issues in existing approaches.

[0215] Limiting Data: The most common datasets for voice-to-lip models are the BIWI and VOCASET datasets. These datasets consist of audio clips of multiple identities along with extremely high-precision tracked meshes of the speakers (i.e., 3D models of the faces). The problem with this is that fine-tuning the model is impossible since a similar quality mesh of the target identity is required.

[0216] Identity Templates: Additionally, since the BIWI and VOCASET datasets were created in a "clean" (read impractical) setting, they provide a grid of identity templates from which predictions are made. This again limits the ability to fine-tune new identities, since it is impractical to obtain this grid.

[0217] Lip Style: Finally, FaceFormer learns the "style" of each speaker through an embedding layer that takes as input a one-hot embedding keyed by the identity of the lips and voice in the training set. This choice constrains the model to predict lips based on one of the identities in the training set. Using another individual's lips to make predictions can be problematic because the geometry of individual lips is unique.

[0218] As described in this paper, a voice-to-lip model can be trained to predict an individual's lip keypoints based on any provided video with image frames capturing the individual speaking. The benefit of processing video directly is that the keypoints extracted for training purposes can be extracted from any video, enabling fine-tuning for target footage.

[0219] The voice-to-lip model is configured to extract lip key points, audio, and an identity template from a reference video corresponding to an individual. The reference video is labeled with the identity of the individual. The identity template can be a 3D mesh of the individual's lips. This data is then smoothed to reduce noise (remove high frequency noise) before being used for training. In some embodiments, the voice-to-lip model can extract 40 key points from the lips, and 21 key points describing the jaw line, for a total of 61 key points. It should be understood that different numbers of facial key points (e.g., lip key points) can be extracted.

[0220] Fig.14 14 is a block diagram illustrating a voice-to-lip model having a data creator model 1402 configured to extract lip key points 1404, audio 1406, and identity templates 1408 from a reference video 1410 corresponding to an individual identity 1412.

[0221] The identity template may be a 3D mesh of the individual's lips. A synthetic lip or chip keypoint data set adjusted for the individual may be determined based on deviations from a particular identity template 1408 .

[0222] The identity template 1408 may be extracted in a variety of ways. For example, the identity template 1412 may be based on a "still" pose image (in Fig.14 The idea of ​​this "still" pose image is closely related to the BIWI and VOCASET datasets that provide similar identity template grids. However, this approach is limited because "still" pose images may not be usable for new identities. In the present invention, the identity template 1408 for the individual is generated from the average of all key points extracted from the reference video corresponding to the individual. Providing a single identity created by the average of all extracted lips not only performs better, but also solves the problem of deciding which template to predict δ from.

[0223] Finally, due to the importance of lip style (personality) to the generation process, the present method attempts to remove the reliance on the "one-hot" identity specification present in FaceFormer. Instead of restricting the model to generate a "one-hot" recognition of lips based on the style of identities in the training set, the present invention attempts to learn the speaker "style" from a given lip sequence of an individual. For example, the model can sample a sequence of key points from another data set instance of a given identity. This key point sequence can then be used to inform the speaker's style. The idea is that by swapping the sampled sequence for each sample (but ensuring that it is from the same identity), the "style embedding" layer will be able to adapt to new identities at test time.

[0224] Fig. 13B An example sequence sampler 1350 is shown in accordance with some embodiments. The sequence sampler may contain multiple mouth shapes based on identity 1302, frame 1304, and video 1306.

[0225] The voice-to-lip model mapping is fine-tuned for the new identity by extracting lip keypoints and sounds from the original video and tuning the "style encoder" specifically for the new target identity. Once fine-tuned, the voice-to-lip model can generate lips from arbitrary audio in the style of the target identity.

[0226] Lips to Image Network

[0227] Fig.15A is an example lips-to-image network 1500 according to some embodiments.

[0228] As depicted, the lip-to-image network 1500 comprises a first stage and a second stage. In the first stage of the network, a mask frame 1502 and keypoint codes 1512 learned from the lip and jaw geometry (see explanation below) are received to produce a rough estimate or intermediate result 1504 of a reconstructed frame. The reconstructed frame may miss some details. In the second stage of the network, the appearance codes and the intermediate result 1504 from the previous stage are received to produce a detailed reconstruction as an output sequence 1506. The detailed reconstruction may contain details previously missed in the intermediate result 1504.

[0229] The lip-to-image network 1500 can include a transformer encoder to encode the lip geometry of the target lip and jaw keypoints. This encoding of the target geometry is called a "keypoint code" 1512. As depicted, the keypoint code 1512 can be passed to both the personal codebook 1508 and the first stage of the network through an adaptive group-wise normalization layer. Note that the appearance code can be learned per ID. To obtain the appearance code, a personalized codebook 1508 can be learned for each identity. Then, a set of coefficients or weights 1510 can be estimated based on the keypoint codes, which are multiplied by the feature vector of the codebook to produce the final appearance code.

[0230] For these two stages, a UNET network similar to the DDPM structure can be used.

[0231] To make the lip geometry and texture believable, the network 1500 can be first trained on an initial dataset of individual speakers, and then fine-tuned to fit videos of a single actor speaking. This fine-tuning process biases the network 1500 toward generating lip geometry and texture that is specific to the target actor being dubbed. Note that a personal codebook can first be learned on the entire dataset, and then fine-tuned for identity.

[0232] In some cases, the lips in the input frame may be closed, and the lips in the output frame may be open. In some cases, the lips in the input frame are open, and the lips in the output frame are closed. Fig.16 As shown, the system implements process 1600 to address these situations, the input frame 1602 can be masked by a mask region 1606 according to the maximum area covered by the jaw to reduce potential texture artifacts in the detailed reconstruction frame 1604. The mask frame can define the repair area 1608 used to generate at least one of the coarse reconstruction frame (i.e., intermediate result) and the detailed reconstruction frame. This is crucial, otherwise a double chin or some artifacts in the texture may appear.

[0233] The lips-to-image network 1500 can utilize various losses. A number of example losses are described below, for example, using first, second, third, and / or fourth losses that can be used together in various combinations to establish an overall loss function for optimization.

[0234] The first loss can be a mean squared error loss, which measures the squared difference in pixel values ​​between the ground truth and the output image of the network 1500. The second loss can be a learned perceptual image patch similarity (LPIPS) loss, which measures the difference between patches in the ground truth image relative to the output image of the network 1500. The third loss can be a "height-width" loss, which measures the difference between the lip opening between the ground truth and the network output. The neural network can be used as a differentiable module to detect key points on the output lips and the ground truth, and compare the difference in the lip key points (i.e., the fourth loss). Finally, a lip sync expert discriminator is used to correct the synchronization between the audio and the output.

[0235] The lip-to-image network works directly on a generator network such as, but not limited to, StyleGAN. The method learns a set of codes that represent visemes, and then for each lip shape, the network produces a set of coefficients that, if multiplied by the code, produce any lip shape.

[0236] This expressiveness allows for a given point in the latent space representing a face to move along a certain direction to cause a local, meaningful edit to the face. For example, moving in one direction might turn black hair blond, and moving in another direction might turn lips into a smile.

[0237] The problem that the method aims to solve is to find directions in the generator (e.g., StyleGAN) latent space that represent different lip movements of a person when speaking. Applicants solve this problem by realizing that human lip movements can be roughly divided into a finite number of groups, which, if learned, can be combined to create any arbitrary lip shape.

[0238] In some embodiments, the system may include a machine learning architecture having only a single UNET network. Fig. 15B is another example lips-to-image network 1550 according to some embodiments.

[0239] As depicted, the lip-to-image network 1550 contains only a single UNET network. The UNET network model receives and processes the mask frame 1502, the keypoint codes 1512 learned from the lip and jaw geometry, and optionally the appearance code to produce the final reconstructed frame 1506, thereby skipping the Fig.15AIn some embodiments, the appearance code is not used to generate the output sequence 1506.

[0240] The lip-to-image network 1550 can contain a transformer encoder to encode the lip geometry of the target lip and jaw keypoints. This encoding of the target geometry is called a "keypoint code" 1512. As depicted, the keypoint code 1512 can be passed to both the personal codebook 1508 and the network through an adaptive group normalization layer. Note that the appearance code can be learned per ID. To obtain the appearance code, a personalized codebook 1508 can be learned for each identity. Then, a set of coefficients or weights 1510 can be estimated based on the keypoint codes, which are multiplied by the feature vector of the codebook to produce the final appearance code.

[0241] Fig.17 An example architecture 1700 of a model is shown, illustrating the various steps of face generation.

[0242] In the first step, the system changes the lip shape of each frame of a given video to a canonical lip shape and encodes the image into the StyleGAN latent space using E4E. Normalization of the lip shape can be done in several ways. One method is to mask the lower area of ​​the face similar to the UNet method and train the encoder from scratch to learn the canonical lip shape. Another method is to apply the Gangealing process 1702 to each frame, take the average of the frames in the condensed space, and paste the lower part of the average image back into each frame. The advantage of this method over the mask method is that it avoids training the encoder from scratch using a pre-trained E4E encoder and does not miss details of the lower facial area due to the mask.

[0243] In a second step, the system is adapted to learn edit directions that change the canonical lip shape to an arbitrary lip shape represented by a set of lip keypoints 1704. This is done by representing different lip movements in the StyleGAN space as linear combinations of a set of learnable orthogonal directions 1708. Each of these directions should represent a change from a canonical lip shape to a viseme, and combinations of these visemes can be used to generate any arbitrary lip shape. Applicants frame the problem of learning these directions as a reconstruction problem, where the network directly optimizes the directions by learning to change the canonical lip shape to the correct lip shape for each frame during training.

[0244] More precisely, the applicants first extract key points 1704 from the face in a given frame and pass them through the MLP to determine the coefficients of the linear combination. Then, the system orthogonalizes the directions using the Gram-Schmidt method and calculates the linear combination. Finally, the system adds the combination to the canonical latent code given by the E4E encoder.

[0245] In the last step, the system passes the resulting latent code from the previous step to the pre-trained StyleGAN generator and outputs the image 1710. The training process is supervised by the L2 and LPIPS losses between the generator output and a given frame.

[0246] To lip dub a given video, in this embodiment, the system may obtain a stream of lip keypoints from the Voice2Lip network and pass it into the frame, rather than extracting the lip keypoints from the frame.

[0247] Fig.18A and Fig.18B 1800 is a process flow diagram (including sub-process 1800A and sub-process 1800B) mapped across two pages according to some embodiments, which illustrates a method for generating an output video using a machine learning method.

[0248] exist Fig.18A , process 1800A is shown to demonstrate how to train and fine-tune an autoregressive model for inferring lip shape from audio.

[0249] Initial LipFormer Process and Method Steps

[0250] Subprocess 1800A begins with training data, and in this method, examples related to a system for forming lips (e.g., LipFormer) are described. The training data for LipFormer can be video recordings with a single speaker speaking into a camera in the field of view. This data can be collected by recording an internal employee speaking predefined sentences for a range of visemes (lip shapes).

[0251] Once this data is collected, the system can begin the lip former pre-processing process. For each video in the dataset, the process may include:

[0252] 1. Detect faces and key points for each frame in the video

[0253] 2. Project 2D pixel space key points into canonical 3D using Procrustes analysis

[0254] a. Normative representation moves all keypoints to a common space regardless of the position of the face in the image

[0255] 3. Extract audio from video

[0256] 4. Write audio, key points and identities into the data set for training

[0257] a. For simplicity, the identity markers are on the video

[0258] The machine learning model LipPuppet is trained to generate lip key points given only audio and an identity. The system can train LipPuppet on a "global" pool of data, and then fine-tune the model to any new identity in subprocess 1800B. Without fine-tuning, the global model can produce lip shapes that match any of the training identities, but will not capture the details of a specific unseen identity.

[0259] LipPuppet can be used directly without fine-tuning, but Lip will not capture the intricate details of each unique identity. If data is available for fine-tuning, the following process can be used to tune LipPuppet to the identity of interest.

[0260] 1. Process the given identity lens according to "Data and Preprocessing"

[0261] 2. Load the global LipPuppet model

[0262] 3. Initialize the "style" embedding to be learned

[0263] 4. Optimize the "style" embedding layer and freeze (or not freeze) other LipPuppet layers

[0264] 5. Fine-tune the training data until convergence

[0265] The goal of fine-tuning is to learn the "style" of any speaker not in the training set.

[0266] Once fine-tuned, the inference pipeline can consist of:

[0267] 1. Load a LipPuppet model for a specific identity (or globally)

[0268] 2. Split the audio segment into segments of length N milliseconds (LipPuppet has a maximum sequence length)

[0269] 3. Overlap audio segments by K milliseconds

[0270] 4. Pass each segment forward

[0271] 5. Connect the generated lip key points and average them in the overlapping area

[0272] The lip keypoints are now ready for lip2face. Note that not discussed here is the "Dubbing Manager", which can be configured to apply filtering to the lip keypoints before passing them to lip2face. This filtering helps with transitions between silence in the dubbing track and moments where the lip shapes match between the source and the dubbing.

[0273] The initial Lip2Face pipeline and method steps are described, involving how the system trains and fine-tunes a model for filling in lip texture given a lip shape and mask input frame. Lip2Face data requirements are similar to LipFormer, except that no audio is required. Lip2Face may require the original frame as well as its extracted keypoints.

[0274] 1. Detect faces and 2D keypoints for each frame in the video

[0275] 2. Crop and rotate the image to the face

[0276] a. Rotate to keep the eyes in a common position

[0277] 3. Generate a mask to cover the mouth area

[0278] a. It can be from the tip of the nose downwards, or along the contour of the chin

[0279] b. May also include the nose area and extend to cover the mouth area

[0280] 4. Write out images, crops, masks and keypoints

[0281] In some embodiments, masking from the tip of the nose downward (i.e., not including the nose) can enable information smuggling during training, causing the machine learning model to over-focus on smile lines or cheeks in the input frame. Therefore, a mask that includes additional regions, such as a mask that includes the nose region, may bring technical and significant improvements to the results.

[0282] The presence of certain facial expressions, such as smile lines, limits the flexibility of the machine learning model in locating lips on the face. This is because smile lines are taken into account (i.e., interpolated) when generating new lip shapes suggested by the machine learning model's neural network. Therefore, during training, the network needs to balance both the desired lip shape (suggested by the lip geometry condition) and the constraints imposed by the smile lines present in the input video.

[0283] For example, a person in an input video may have smile lines. In fact, a person cannot have smile lines and make an "ooo" mouth shape at the same time. During training, the machine learning model may inadvertently receive cues about lip shape from information hidden in the smile lines, resulting in information smuggling. This may cause the model to over-focus on the smile lines or cheeks during inference, resulting in inaccurate lip shape predictions. Therefore, masking one or more regions of the face that have a high correlation with lip shape improves machine learning model performance.

[0284] In some embodiments, the input image is used only for texture and the lip keypoints are used only for mouth shape.

[0285] Lip2Face is trained to "fill in" a given mask input image using given lip keypoints for that frame and optionally an identity. The system trains Lip2Face on a "global" pool of data, and can then fine-tune it to better capture texture based on a specific identity. In another embodiment, the model can be used without fine-tuning, but if data is available, fine-tuning will improve the results.

[0286] Fine-tuning

[0287] As mentioned above, Lip2Face can be used directly without fine-tuning, but the texture of the generated lips may not be of high quality. If data is available, Lip2Face can be fine-tuned using the following process.

[0288] 1. Process the given identity lens according to "Data and Preprocessing"

[0289] 2. Load the global Lip2Face model

[0290] 3. Initialize “style” embedding

[0291] 4. Optimize "style" embedding, freeze other lip2face layers (or not)

[0292] 5. Fine-tune the training data until convergence

[0293] 6. The goal of fine-tuning lip2face is to learn a “style” embedding that represents this new unseen identity.

[0294] Lip2Face's inference process can use keypoints generated by LipPuppet. Lip2Face can also use keypoints extracted directly from video footage, which simplifies the process. The following process is used to create a new dubbed frame from the lip keypoints.

[0295] 1. Process the video to be dubbed according to "Data and Preprocessing"

[0296] 2. Load the driver lip key points generated by LipPuppet

[0297] 3. Align the actuation keypoints with the extracted “source” lips using the source lip transformation from pixels to canonical space

[0298] 4. (Optional) Least squares lip keypoints lip personalization to match source

[0299] If LipPuppet fine-tuning is not done or is unsuccessful, you can use

[0300] 5. Pass each masked source frame forward, replacing the keypoints with the loaded and aligned driver keypoints

[0301] 6. Create a video based on the inference results and add the dubbing audio as the soundtrack.

[0302] At the end of this process, the output is a dubbed video. The Dubbing Manager process can be used again here to replace unwanted frames (for example, when there is a character laughing or the dubbing and the original file are silent, these frames can be removed).

[0303] Fig.19 is an example schematic block diagram 1900 of components of a system for performing lip dubbing according to some examples. In the schematic block diagram 1900, a set of computational processes are shown, including different machine learning models and program code execution blocks, which can be implemented in the form of modular computer programs stored on non-transitory computer readable memory.

[0304] Fig. 20 An example computing process flow 2000 is shown that may be used as part of a processing pipeline in a commercial implementation. Fig. 20 , the diagram illustrates steps that can be performed in parallel and in series according to various embodiments described herein, such that computational input is received, a model is trained, and the trained model is deployed to automatically generate output.

[0305] The system may be implemented as a dedicated machine, such as a dedicated computing device that may be part of or operate as a computer server. For example, a rack-mounted device that may be used in a data center for a specific purpose: receiving input video over a message bus as part of a processing pipeline to create output video. The dedicated machine is used as part of a visual effects post-production computing method, for example, editing after the initial material is produced. Editing may include the integration of overlays or introduced computer graphics elements to replace portions of live action footage or animation, and such editing may be computationally intensive.

[0306] A special purpose machine may be instructed according to a set of machine-interpretable instructions that cause a processor to perform the steps of a computer-implemented method. The set of machine-interpretable instructions may be attached to a physical non-transitory computer-readable medium as an article of manufacture, such as a tangible physical storage medium (e.g., an optical disk, a solid-state drive, etc.) that may be provided to a computer server or computing device to be loaded or executed with various programs.

[0307] In the context of the disclosed method, a pipeline receives an input for post-processing, which may include a video data object and a target audio data object. The system is configured to generate a new output video data object that effectively replaces certain regions (such as regions in the mouth region). The target audio data object may first be decomposed into time-stamped audio tokens, which are mapped to phonemes and then mapped to corresponding visemes. Effectively, each time-stamped audio token may represent a mouth shape or mouth movement corresponding to the target audio data object.

[0308] Since the original video has speech in the original language, it is necessary to adjust the individual's mouth movements and / or facial movements in the output video in an automatic manner to match the target audio data object (e.g., the target language soundtrack). As described herein, it is difficult and impractical to perform this process manually, and what is proposed herein is a machine learning method that attempts to automatically generate a replacement video.

[0309] The first instance of the special purpose machine may include a server configured to generate a replacement output video object based on a parameter instruction set when controlling the operation of the machine learning network, the parameter instruction set being the disentangled expressions and gestures. For example, the parameter instruction set may be based on a specific viseme that corresponds in time to a new mouth movement at a specific point in time, the new mouth movement corresponding to a target mouth movement in a target language of a desired output audio of the output video object. Optionally, the parameter instruction set may be extended with additional parameters representing residual parameters.

[0310] In this example, the machine learning network has two subnetworks, the first subnetwork is a voice-to-lip machine learning model, and the second subnetwork is a lip-to-image machine learning model. In this example, the two models interact together to reconstruct the frame to create a new output video data object. The two models can be used together in a coarse / fine reconstruction process, where an initial coarse frame can be refined to create a fine frame. During the reconstruction process, the models work with a mask frame in which inpainting can be performed to replace specific portions of the image frame only in areas according to the mask frame (e.g., only on the masked portion).

[0311] In some embodiments, the output may be instructions for repair, which may be provided to downstream systems, or in other embodiments, the output may be a replacement region of a masked portion or an entire replaced frame, depending on the configuration of the system. The pipeline computation component may receive a replacement output video or replacement frame, and in other embodiments, may evaluate the quality control of these frames or their video portions, for example, by indicating whether the frame or video portion is approved / unapproved. If the frame / video portion is unapproved, in other embodiments, the system may be configured to regenerate the particular portion, and the disapproval may be used as further training of the system. In some embodiments, an iterative process may be performed until there are no unapproved portions and all portions or frames have passed the quality control process before providing the final output video data object to the next step in the post-processing pipeline.

[0312] The post-processing pipeline can have multiple processors or systems operating in parallel. For example, a video can be received, and the video is in the original language, such as a video in French. It may be desirable to have an audio track in Spanish, English, German, Korean, Chinese, Japanese, Malaysian, Indonesian, Swahili, etc. For example, each audio track in these target audio tracks can be obtained by local voice talents, computer sound synthesis using a translation program, etc. The system can be tasked in post-production to create multiple videos in parallel, wherein the mouth is modified to match each audio track in these target audio tracks. Then, each generated video can go through a quality control process until a reviewer (e.g., a reviewer system or a human reviewer) is satisfied with the output.

[0313] This paper proposes variations of the computational architecture. For example, in the example embodiment, a single U-net is utilized which has shown strong performance in experimental analysis.

[0314] Variants of the masking method are also proposed, for example, an improved mask that extends the masking region to the nose region instead of just under the nose, which is also found to show strong performance in experimental analysis.

[0315] Applicants note that the described embodiments and examples are illustrative rather than restrictive. Actual implementation of features may incorporate a combination of some or all of the aspects, and the features described herein should not be considered as an indication of future or existing product plans.

[0316] Although the embodiments have been described in detail, it should be understood that various changes, substitutions and modifications may be made herein without departing from the scope. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods and steps described in this specification.

[0317] As will be readily understood by those skilled in the art from this disclosure, currently existing or later to be developed processes, machines, manufactures, compositions of matter, components, methods, or steps that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein may be utilized. Accordingly, the appended embodiments are intended to include within their scope such processes, machines, manufactures, compositions of matter, components, methods, or steps.

[0318] It will be understood that the examples described and illustrated above are intended to be exemplary only.

Claims

1. A computer-implemented system for generating an output video of a lip dubbing, the system comprising: processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: Receiving an initial video data object V comprising a plurality of image frame data objects I; receiving a time-synchronized set of viseme parameters corresponding to a target audio data object A'; and The initial video data object V and the time-synchronized viseme parameter set are processed using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized viseme parameter set.

2. A system according to claim 1, wherein the time-synchronized viseme parameter set corresponding to the target audio data object A' is a parameter set identified using multiple encoder networks, and the multiple encoder networks are suitable for disentangling expressions e and postures p according to multiple image characteristics.

3. A system according to claim 2, wherein the multiple encoder networks are iteratively trained to disentangle expression parameters e, posture parameters p and residual parameters r from the multiple image frame data objects I.

4. A system according to claim 1, wherein the processor is configured to: extract an initial expression / viseme code set corresponding to a spectrogram segment in an initial audio data object A corresponding to an initial video data object V.

5. The system of claim 4, wherein the initial video data object V is used to populate a viseme library of available visemes based on the initial expression / viseme code set.

6. A system according to claim 5, wherein the processor is configured to compare the visemes identified in the target audio data object A' with the viseme library of available visemes based on the initial expression / viseme code set to identify a subset of visemes that are not in the viseme library and need to be synthesized.

7. The system of claim 6, wherein the processor is configured to generate new expression / viseme codes corresponding to the subset of visemes not in the viseme library.

8. The system according to claim 1, wherein the processor is configured to generate the output video data object V' by replacing at least one image frame data object of the plurality of image frame data objects I in its entirety with a replacement image frame data object I'.

9. A system according to claim 1, wherein the processor is configured to generate the output video data object V' by replacing a portion of at least one image frame data object among the multiple image frame data objects I with a corresponding portion of the replacement image frame data object I', wherein the portion corresponds to a mouth area.

10. The system of claim 1, wherein the processor is configured to iteratively generate the output video data object V' for each character present in the initial video data object V.

11. The system of claim 1 , wherein the machine learning network comprises a generative adversarial network.

12. A computer-generated method for generating an output video, the method comprising: Receiving an initial video data object V comprising a plurality of image frame data objects I; receiving a time-synchronized set of viseme parameters corresponding to a target audio data object A'; and The initial video data object V and the time-synchronized viseme parameter set are processed using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized viseme parameter set.

13. The method of claim 12, wherein the time-synchronized viseme parameter set corresponding to the target audio data object A' is a parameter set identified using a plurality of encoder networks adapted to disentangle expressions e and gestures p from a plurality of image characteristics.

14. The method according to claim 13, wherein the multiple encoder networks are iteratively trained to disentangle expression parameters e, posture parameters p and residual parameters r from the multiple image frame data objects I.

15. The method according to claim 12, further comprising: An initial expression / viseme code set corresponding to a spectrogram segment in an initial audio data object A is extracted from an initial audio data object A corresponding to an initial video data object V.

16. The method of claim 15, wherein the initial video data object V is used to populate a viseme library of available visemes based on the initial expression / viseme code set.

17. The method according to claim 16 further comprises comparing the visemes identified in the target audio data object A' with the viseme library of the available visemes based on the initial expression / viseme code set to identify a viseme subset not in the viseme library, and generating new expression / viseme codes corresponding to the viseme subset not in the viseme library.

18. The method according to claim 12, further comprising generating the output video data object V' by replacing at least one image frame data object of the plurality of image frame data objects I in its entirety with a replacement image frame data object I'.

19. The method according to claim 12 further comprises generating the output video data object V' by replacing a portion of at least one image frame data object of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to a mouth area.

20. The method of claim 12, further comprising iteratively generating the output video data object V' for each character present in the initial video data object V.

21. A system for generating an output video corresponding to a target set of audio instructions, the system comprising: processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: instantiating a voice-to-lip machine learning model architecture configured to encode a set of input audio associated with an individual into audio tokens that are converted into a set of synthesized lip or chip keypoint data tuned for the individual; and Instantiates a lips-to-image machine learning model architecture configured to: receiving a target frame and a set of keypoint code data learned from lip or jaw geometry in the target frame and generating a coarsely reconstructed frame; receiving an appearance code based on at least the synthesized lip or chip keypoint data set; and A fine reconstructed frame is generated using both at least the coarse reconstructed frame and the synthesized lip or chip keypoint data set.

22. A system according to claim 21, wherein at least one of the voice-to-lip machine learning model architecture and the lip-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

23. The system of claim 21 , wherein at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is fine-tuned during inference on the individual.

24. The system of claim 21, wherein the voice-to-lip machine learning model architecture is trained to predict lip key points of an individual based on any provided video having image frames capturing the individual speaking.

25. The system of claim 21, wherein the set of synthesized lip or chip keypoint data adjusted for the individual is determined based on deviations from an identity template.

26. The system of claim 25, wherein the identity template of an individual is generated based on an average of all extracted key points of reference videos corresponding to the individual.

27. A system according to claim 21, wherein the target frame is a mask frame that is masked based on a maximum area that the jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, and the mask frame defines a repair area used to generate at least one of the coarse reconstruction frame and the fine reconstruction frame.

28. The system of claim 21, wherein the appearance code is a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients estimated based on at least the keypoint code data set for scaling feature vectors.

29. The system of claim 21, wherein the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip openness between a ground truth value and at least one of the coarsely reconstructed frame and the finely reconstructed frame.

30. The system of claim 21, wherein the processor is further configured to: identify segments in an input video V having input audio A and a target output video V' having output audio A', wherein lip shapes represent different visemes, and generate, for only each frame of the identified segment of the input video V, a corresponding finely reconstructed frame corresponding to the viseme of the output audio A'.

31. A computer-implemented method for generating an output video corresponding to a target set of audio instructions, the method comprising: instantiating a voice-to-lip machine learning model architecture configured to encode a set of input audio associated with an individual into audio tokens that are converted into a set of synthesized lip or chip keypoint data tuned for the individual; as well as Instantiates a lips-to-image machine learning model architecture configured to: receiving a target frame and a set of keypoint code data learned from lip or jaw geometry in the target frame and generating a coarsely reconstructed frame; receiving an appearance code based on at least the synthesized lip or chip keypoint data set; and A fine reconstructed frame is generated using both at least the coarse reconstructed frame and the synthesized lip or chip keypoint data set.

32. The method of claim 31 , wherein at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

33. The method of claim 31 , wherein at least one of the voice-to-lips machine learning model architecture and the lips-to-image machine learning model architecture is fine-tuned during inference on the individual.

34. The method of claim 31 , wherein the voice-to-lip machine learning model architecture is trained to predict lip key points of an individual based on any provided video having image frames capturing the individual's speech.

35. The method of claim 31 , wherein the set of synthesized lip or chip keypoint data adjusted for the individual is determined based on deviations from an identity template.

36. The method of claim 35, wherein the identity template of an individual is generated based on an average of all extracted key points of reference videos corresponding to the individual.

37. A method according to claim 31, wherein the target frame is a mask frame that is masked based on the maximum area that the jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, and the mask frame defines a repair area used to generate at least one of the coarse reconstruction frame and the fine reconstruction frame.

38. The method of claim 31, wherein the appearance code is a personalized codebook learned for the individual and the appearance code is used to establish a set of coefficients estimated based on at least the keypoint code data set for scaling feature vectors.

39. The method of claim 31 , wherein the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip openness between a ground truth value and at least one of the coarsely reconstructed frame and the finely reconstructed frame.

40. The method of claim 31, further comprising: Segments in an input video V with input audio A and a target output video V' with output audio A' are identified, where lip shapes represent different visemes, and for each frame of the identified segment only of the input video V, a corresponding finely reconstructed frame corresponding to the viseme of the output audio A' is generated.

41. A non-transitory computer readable medium storing a set of machine interpretable instructions which, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 12 to 20.

42. A non-transitory computer readable medium storing a set of machine interpretable instructions which, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 31 to 40.

Images