System and method for improving lip dubbing

Abstract

A computer-implemented system is described in various embodiments herein, the system comprising: a processor; a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: receive an initial video data object V comprising a plurality of image frame data objects I; receive a time-synchronized set of visual-inertial parameters corresponding to a target audio data object A'; and process the initial video data object V and the time-synchronized set of visual-inertial parameters using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized set of visual-inertial parameters.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 301,947, filed January 21, 2022, and U.S. Provisional Patent Application No. 63 / 426,283, filed November 17, 2022, which are incorporated herein by reference in their entirety. Technical Field

[0003] Embodiments of this disclosure generally relate to the field of machine learning for visual effects, and more specifically, embodiments relate to systems and methods for improving the manipulation of lip movements in videos or images, for example, to match dubbed video footage in a target language. Background Technology

[0004] The amount of content available on television is increasing rapidly. Foreign films are becoming more popular in English-speaking countries, and international streaming platforms are making it easier for non-English speakers to access English-language content.

[0005] To better appeal to audiences who speak languages ​​different from the film in question, it is desirable to translate the script and then dub it. However, individual audio dubbing does not match the speaker's lip movements and can lead to timing inconsistencies. Therefore, manipulating lip movements to match the dubbed film in any given language is useful. However, manual manipulation is practically impractical given the enormous effort required per frame. Summary of the Invention

[0006] Therefore, there is a clear and growing need for automated systems and methods that can manipulate V to obtain a video V' based on the audio A' of the spoken language L', such that the lips in the video V' match the audio A'. However, this presents a challenging technical problem because videos are typically of high quality and high resolution, such as 4K or higher, and subtle mismatches or slight noise can produce noticeable artifacts that should ideally be corrected and removed.

[0007] Various embodiments described herein describe computer-implemented methods in which a specially configured machine learning model architecture is trained and deployed to receive input data objects V (a data representation of the raw video consisting of raw video frame objects I) and A (a data representation of the raw audio). In some embodiments, a data object A' (a data representation of the target audio) is provided, while in other embodiments, said data object A' is generated (e.g., a machine translation step may occur to generate A').

[0008] The machine learning model architecture may include a generator network, which may include, but is not limited to, generative adversarial networks (such as StyleGAN), trained as a generator to track relevant image features and recombine them based on a set of input hyperparameters into an “illusion” or machine-generated output image or image portion. The machine learning model architecture in the proposed method generates generated image frame data objects I', where I' can be the entire image or an image portion specifically related to the mouth region of a character on screen (e.g., a human or animated character).

[0009] The parameters provided to the machine learning model architecture for generating I' are modified relative to I, such that new parameter definitions for the new viewpoints at corresponding timestamps are used to generate I'. Therefore, the machine learning model architecture is trained to computationally "redraw" the character's mouth region based on a set of new target timestamped viewpoints that can be extracted from A'. Then, when recombining or generating the output video data object V', some or all of the image frame data object I is replaced with the generated image frame data object I'.

[0010] A view position can represent a set of several sounds that appear identical, such as when lip-reading. In video processing, a view position can contain data representing the location (or pixels of) of one or more areas of the face (e.g., the mouth area and surrounding area) when a set of several sounds that appear identical are spoken. For example, in English, the words "pet" and "bet" are pronounced differently. However, when viewed visually (without sound), the mouth areas of the people speaking these two words appear visually identical.

[0011] This method can be used, for example, to computationally automate previous manual lip-syncing / redrawing exercises, and the problem that is associated with previous methods is lip-syncing, where the male / female actor speaking the target language must match the syllables with the original lip movements (resulting in inconsistencies in the timing or script of the target language) or the on-screen lip movements do not correspond correctly to the audio of the target language (mouth movement but no speech, or no movement but the character is speaking).

[0012] Another embodiment includes the use of a system for animation post-processing, where the output video is generated directly based on a time-synchronized set of viewpoints, rather than drawing the initial face or mouth in the original video, and for example, drawing the mouth or facial regions directly as part of a rendering step. This reduces the effort required to prepare the initial video for input.

[0013] In yet another embodiment, viewpoint data is provided to a machine learning architecture that includes a machine learning network (e.g., a generative network) that generates video without the original input video and generates a completely “illusionary” video with the correct mouth shape and mouth movement corresponding to the target audio track, based on a set of instructions or storyboard data objects.

[0014] Two technical challenges were identified in adapting the machine learning data model architecture for the proposed use: (1) a computational method is needed to decouple facial expressions (viewpoints and lip shapes) from other features (such as pose, lighting, and overall texture); and (2) the system needs to be adapted to realistically synthesize missing viewpoints (e.g., useful when the correct viewpoints are not present in the input video, such as when the input video is short).

[0015] Untangling is used to identify and define a set of parameters that control the operation of machine learning networks (e.g., generative networks). Identifying and defining a set of parameters is not straightforward because it is difficult to control specific output aspects of a machine learning model, many of which are learned iteratively in a "latent space" that is difficult to interpret or study. Untangling is a technique that allows a machine learning architecture system to manipulate certain attributes of an image (e.g., properties of the generated image related to the mouth region, making it possible to manipulate the mouth region to match new view positions) without changing other aspects. Untangling can be used as a first step in computationally learning how to control a machine learning model, which is then used to generate new images (e.g., new images with modified mouth regions). Untangling methods can produce controllable parameters for machine learning networks, which can be generative networks such as StyleGAN.

[0016] For example, by using the output of an unwrapping method, a machine learning architecture system can be configured to preserve the pose, lighting conditions, or identity of an image and modify only the facial expression. To do this, the machine learning architecture system learns individual codes or representations for each aspect of the image, such as identity, facial expression (i.e., view position), and pose, and the system then maps these to a single code that can generate an image with the same facial expression, identity, and pose as the input code.

[0017] When the latent space is generated or maintained by an autoencoder or a mapper in a generative network such as StyleGAN, "code" refers to a low-dimensional vector that can be fed into a generator / decoder network to generate an image during inference. Such codes are learned and are low-level representations of high-dimensional data, such as images.

[0018] Latent codes can be represented by vectors in, for example, a multidimensional space, which can be a high-dimensional space. When the latent space is unwrapped, directions in the latent space are associated with corresponding characteristics or aspects of the image, such that when a vector value moves along said direction, the corresponding characteristics or aspects of the image change or are modified accordingly. Such directions in the latent space, which can be represented by vectors (e.g., latent codes), can only change the age, expression, or posture of a character within the image.

[0019] Therefore, embodiments of the system architecture described herein are configured to utilize the untangling properties of latent space to find the direction or vector of "expressions" (e.g., mouth shape or lip shape) within an image. If the image remains unchanged, movement along this direction can generate all possible expressions while preserving other aspects (e.g., pose, lighting, identity).

[0020] Therefore, a single code, or a single latent code, is a single "point" in the latent space. This "point" represents a specific identity in an image, with a specific pose and a specific expression.

[0021] In some embodiments, the process performed by the system may include training a neural network that can individually learn poses, identities, or expressions as codes, indices, or sets of 2D or 3D points. To ensure that each code is used correctly, the system can replace certain codes (e.g., poses) of an image with codes of another image (another pose), and certain codes (e.g., poses) of an image can be replaced with codes of another image (another pose), and supervised pairwise training can ensure that all aspects except those corresponding to the modified code (i.e., pose) remain intact.

[0022] Viewpoint synthesis may involve generating 3D models with good realism (e.g., within an acceptable range defined by various visual metrics) in different poses and viewpoints. For this purpose, a virtual camera can be placed on a sphere with a textured 3D model, which may be located at the origin. Several images can be presented in different poses and viewpoints. These synthesized images can then be fed into a generative image-to-image translation model to make them more realistic (e.g., optimizing a specific metric representing realism or acting as a proxy for realism). For example, this generative model could be a mapper that receives the synthesized images and maps them to a distribution of real images. Instances of such networks could include properly trained CycleGANs or StyleGANs (such as pSp).

[0023] The method described in this paper is actually implemented using a computer system that maintains a trained generative adversarial model, which is physically implemented on computer memory and storage devices by refining and optimizing the interconnection weights of different hidden features in the latent space through a training process (e.g., an iterative training process).

[0024] In some embodiments, the computer system may include a computer server or a cloud computing system based on distributed resources, and may be used, for example, with or as part of a video generation / translation / rendering process, such that, for example, during post-processing, the input video (animated or real-life) can be processed to generate an output video with audio in various output languages, wherein the mouth area of ​​the output video has been manipulated to match the target audio.

[0025] As described in other embodiments herein, further details are presented in the example system where a two-level network is provided. In this example system, the system is configured to generate output video corresponding to a target set of audio instructions (e.g., for lip-syncing or replacement purposes).

[0026] An optional initial stage can be implemented, whereby the example system in this example embodiment identifies segments in an input video V with input audio A and a target output video V' with output audio A', where lip shapes are represented by different viewpoints, and only for each frame of the identified segment of the input video V, it is configured to generate a corresponding finely reconstructed frame corresponding to the viewpoint of the output audio A'. This is a useful method to reduce the overall computational burden, as it eliminates the need to generate image frames with identical viewpoints. In this initial stage, sound alignment is performed to identify similar viewpoints between sound segments or silent regions (e.g., both are silent). For these segments, a bypass method can be used to simply copy frames from the input video instead of using system-generated frames. In this optional stage, a pipelined approach can be used, where the identified frames / audio segments requiring lip-syncing are fed into a processing pipeline for generation.

[0027] The system may be a physical computing device such as a computer server, and the server may contain a processor coupled to computer memory, such as a server in a cluster of processing servers that can access computing resources.

[0028] As part of the processing, the system can leverage a two-stage machine learning model topology, such as a voice-to-lip model (e.g., a machine learning model architecture) working in conjunction with a lip-to-image model. The voice-to-lip model is a network configured to receive the geometry of the lips and animate it based on the sound. The lip-to-image model is a network configured to receive the personalized geometry of the lips (based on the audio) and each frame requiring dubbing. Each frame to be dubbed is first analyzed to extract the existing lip shape (in some instances, this can be done by generating a mask for masking purposes), and then this area corresponding to a given lip shape (e.g., a masked area) is "filled" based on the personalized geometry of the lips. For example, the filling may include repair. The voice-to-lip and lip-to-image model networks can be pre-trained against many different people or identities ("IDs"), but during inference, they are fine-tuned for each speaker if the video is long enough. The geometry of the face (e.g., lips, chin, and / or jaw) helps the network better predict the final reconstructed image (output dubbing). During testing, various variants were used to address issues related to unrealistic lips, lip position variations, chin stretching, etc. For example, to address chin positioning issues, additional keypoints can be used for chin tracking, and for lip modeling, an improved keypoint extraction model can be applied to obtain a more accurate lip shape. Architectural changes can improve aspects such as "lip adhesion." These issues can be further addressed using weightlessness balancing and other model modifications.

[0029] Identity, or ID for short, is a synonym for a person or role. When one or more machine learning networks are trained, pre-trained, or fine-tuned, the goal of this training, pre-training, or fine-tuning is to preserve or capture the speech texture and style of a specific person or identity.

[0030] The voice-to-lip machine learning model architecture is configured to encode a set of input audio associated with an individual into audio tokens, which are then converted into a set of synthetic lip or chip keypoint data tailored to the individual. The model is instantiated and trained for use during inference. The voice-to-lip machine learning model architecture is trained to predict the individual's lip keypoints based on any provided video containing image frames capturing the individual speaking. The set of synthetic lip or chip keypoint data tailored to the individual is determined based on deviations from a specific identity template. The individual's identity template may be generated, for example, based on the average of all extracted keypoints from a reference video corresponding to the individual.

[0031] The lip-to-image machine learning model architecture is configured to receive a target frame and a set of keypoint code data learned from the lip or jaw geometry in the target frame and generate a coarse reconstructed frame. Then, it receives appearance code based at least on the synthetic lip or chip keypoint data set, and uses at least the coarse reconstructed frame and the synthetic lip or chip keypoint data set to generate a fine reconstructed frame. The model is then instantiated and trained for use during inference.

[0032] The lip-to-image machine learning model architecture may utilize at least a height-width loss function that measures the difference in lip opening between the ground truth and at least one of the coarse reconstructed frame and the fine reconstructed frame.

[0033] The appearance code may be a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients, which is estimated based at least on the keypoint code data set used to scale the feature vector.

[0034] The voice-to-lips machine learning model architecture and the lip-to-image machine learning model architecture can be pre-trained based on multiple different human individual identities and can also be fine-tuned during the inference time on individuals.

[0035] The target frame used for replacement can be a masked frame that is masked based on the maximum area that the jaw can cover in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the masked frame defining a repair area for generating at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0036] The method described in this paper is useful in reducing the overall post-processing work required to generate modified videos based on input video sources (e.g., where lips are replaced in certain segments or image frames). In particular, specific adjustments are proposed that use improved machine learning methods to help improve the accuracy of the generated frames or portions thereof, which can be personalized for one or more individuals.

[0037] Machine learning methods can be used to help systems avoid the "uncanny valley" problem. For commercial implementation, it is important to have sufficient accuracy to avoid arousing suspicion when people watch the generated videos.

[0038] The mechanism can be implemented as a post-processing visual effects pipeline, for example, for television programs, movies, etc.

[0039] According to one aspect, a computer-implemented system for generating an output video for lip-syncing is provided, the system comprising: a processor; and a memory coupled to the processor and storing processor-executable instructions, which, when executed, configure the processor to: receive an initial video data object V comprising a plurality of image frame data objects I; receive a time-synchronized set of view positions parameters corresponding to a target audio data object A'; and process the initial video data object V and the time-synchronized set of view positions parameters using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized set of view positions parameters.

[0040] In some embodiments, the set of view position parameters corresponding to the time synchronization of the target audio data object A' is a set of parameters identified using multiple encoder networks adapted to unwrap expression e and pose p according to multiple image characteristics.

[0041] In some embodiments, the plurality of encoder networks are iteratively trained to unwrap the expression parameter e, pose parameter p, and residual parameter r from the plurality of image frame data objects I.

[0042] In some embodiments, the processor is configured to extract an initial set of facial expression / viewpoint codes corresponding to spectrogram segments in the initial audio data object A from an initial audio data object A corresponding to the initial video data object V.

[0043] In some embodiments, the initial video data object V is used to populate the viewpoint library of available viewpoints based on the initial expression / viewpoint code set.

[0044] In some embodiments, the processor is configured to compare the gaze positions identified in the target audio data object A' with the gaze position library of available gaze positions based on the initial expression / gaze position code set to identify a subset of gaze positions that are not in the gaze position library and need to be synthesized.

[0045] In some embodiments, the processor is configured to generate new expression / viewpoint codes corresponding to the subset of viewpoints that are not in the viewpoint library.

[0046] In some embodiments, the processor is configured to generate the output video data object V' by replacing at least one of the plurality of image frame data objects I with a replacement image frame data object I'.

[0047] In some embodiments, the processor is configured to generate the output video data object V' by replacing a portion of at least one of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to the mouth region.

[0048] In some embodiments, the processor is configured to iteratively generate the output video data object V' for each character present in the initial video data object V.

[0049] In some embodiments, the machine learning network includes a generative adversarial network.

[0050] According to another aspect, a computer-generated method for generating an output video is provided, the method comprising: receiving an initial video data object V comprising a plurality of image frame data objects I; receiving a time-synchronized viewpoint parameter set corresponding to a target audio data object A'; and processing the initial video data object V and the time-synchronized viewpoint parameter set using a machine learning network to generate an output video data object V', wherein an initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized viewpoint parameter set.

[0051] In some embodiments, the set of view position parameters corresponding to the time synchronization of the target audio data object A' is a set of parameters identified using multiple encoder networks adapted to unwrap expression e and pose p according to multiple image characteristics.

[0052] In some embodiments, the plurality of encoder networks are iteratively trained to unwrap the expression parameter e, pose parameter p, and residual parameter r from the plurality of image frame data objects I.

[0053] In some embodiments, the method may include: extracting an initial set of facial expression / viewpoint codes corresponding to spectrogram segments in the initial audio data object A from an initial audio data object A corresponding to an initial video data object V.

[0054] In some embodiments, the initial video data object V is used to populate the viewpoint library of available viewpoints based on the initial expression / viewpoint code set.

[0055] In some embodiments, the method may include: comparing the gaze positions identified in the target audio data object A' with a gaze position library of available gaze positions based on the initial expression / view position code set to identify a subset of gaze positions not in the gaze position library, and generating new expression / view position codes corresponding to the subset of gaze positions not in the gaze position library.

[0056] In some embodiments, the method may include generating the output video data object V' by replacing at least one of the plurality of image frame data objects I with a replacement image frame data object I'.

[0057] In some embodiments, the method may include generating the output video data object V' by replacing a portion of at least one of the plurality of image frame data objects I with a corresponding portion of the replacement image frame data object I', wherein the portion corresponds to the mouth region.

[0058] In some embodiments, the method may include: iteratively generating the output video data object V' for each character present in the initial video data object V.

[0059] According to another aspect, a system is provided for generating an output video corresponding to a target set of audio instructions, the system comprising: a processor; and a memory coupled to the processor and storing processor-executable instructions, which, when executed, configure the processor to: instantiate a voice-to-lip machine learning model architecture configured to encode a set of input audio associated with an individual into audio tokens, the audio tokens being converted into a set of synthetic lip or chip keypoint data tailored to the individual; and instantiate a lip-to-image machine learning model architecture configured to: receive a target frame and a set of keypoint code data learned from the lip or jaw geometry in the target frame and generate a coarse reconstructed frame; receive appearance code based at least on the synthetic lip or chip keypoint data set; and generate a fine reconstructed frame using at least the coarse reconstructed frame and the synthetic lip or chip keypoint data set.

[0060] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lip-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

[0061] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lip-to-image machine learning model architecture is fine-tuned during inference on the individual.

[0062] In some embodiments, the voice-to-lip machine learning model architecture is trained to predict the individual's lip keypoints based on any video provided with image frames capturing the individual speaking.

[0063] In some embodiments, the set of synthetic lip or chip key data points adjusted for the individual is determined based on deviations from the identity template.

[0064] In some embodiments, the identity template of an individual is generated based on the average of all extracted key points from a reference video corresponding to the individual.

[0065] In some embodiments, the target frame is a masked frame that is masked based on the maximum area that the jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the masked frame defining a repair area for generating at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0066] In some embodiments, the appearance code is a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients estimated at least based on the keypoint code data set used to scale feature vectors.

[0067] In some embodiments, the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip opening between the ground truth and at least one of the coarse reconstructed frame and the fine reconstructed frame.

[0068] In some embodiments, the processor is further configured to: identify segments in an input video V having input audio A and a target output video V' having output audio A', wherein lip shapes represent different view positions, and generate corresponding finely reconstructed frames corresponding to the view positions of the output audio A' for each frame of the identified segments of the input video V only.

[0069] According to another aspect, a computer-implemented method is provided for generating an output video corresponding to a target set of audio instructions, the method comprising: instantiating a voice-to-lip machine learning model architecture configured to encode an input audio set associated with an individual into audio tokens, the audio tokens being converted into a synthetic lip or chip keypoint data set adjusted for the individual; and instantiating a lip-to-image machine learning model architecture configured to: receive a target frame and a keypoint code data set learned from the lip or jaw geometry in the target frame and generate a coarse reconstructed frame; receive appearance code based at least on the synthetic lip or chip keypoint data set; and generate a fine reconstructed frame using at least the coarse reconstructed frame and the synthetic lip or chip keypoint data set.

[0070] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lip-to-image machine learning model architecture is pre-trained based on multiple different human individual identities.

[0071] In some embodiments, at least one of the voice-to-lips machine learning model architecture and the lip-to-image machine learning model architecture is fine-tuned during inference on the individual.

[0072] In some embodiments, the voice-to-lip machine learning model architecture is trained to predict the individual's lip keypoints based on any video provided with image frames capturing the individual speaking.

[0073] In some embodiments, the set of synthetic lip or chip key data points adjusted for the individual is determined based on deviations from the identity template.

[0074] In some embodiments, the identity template of an individual is generated based on the average of all extracted key points from a reference video corresponding to the individual.

[0075] In some embodiments, the target frame is a masked frame that is masked based on the maximum area that the jaw can occupy in the target frame to reduce potential texture artifacts in the fine reconstruction frame, the masked frame defining a repair area for generating at least one of the coarse reconstruction frame and the fine reconstruction frame.

[0076] In some embodiments, the appearance code is a personalized codebook learned for the individual, and the appearance code is used to establish a set of coefficients estimated at least based on the keypoint code data set used to scale feature vectors.

[0077] In some embodiments, the lip-to-image machine learning model architecture utilizes at least a height-width loss function that measures the difference in lip opening between the ground truth and at least one of the coarse reconstructed frame and the fine reconstructed frame.

[0078] In some embodiments, the method further includes: identifying segments in an input video V having input audio A and a target output video V' having output audio A', wherein lip shapes represent different view positions, and generating corresponding finely reconstructed frames corresponding to the view positions of the output audio A' for each frame of the identified segments of the input video V only.

[0079] In some embodiments, a non-transitory computer-readable medium is provided that stores a set of machine-interpretable instructions, which, when executed by a processor, cause the processor to perform method steps according to any of the methods described above.

[0080] The system can be implemented as a dedicated machine, such as a dedicated computing device that can be part of a computer server or operate as a computer server. For example, a rack-mounted device can be used for a specific purpose in a data center: receiving input video on a message bus as part of a processing pipeline to create output video. Dedicated machines can be used as part of a computational method for post-production visual effects, such as editing after the initial material has been created. Editing can involve the integration of overlays or introduced computer graphics elements to replace parts of live-action footage or animation, and this editing can be computationally intensive.

[0081] A dedicated machine can be instructed according to a set of machine-interpretable instructions, which causes a processor to perform the steps of a computer-implemented method. The set of machine-interpretable instructions can be attached as an article of writing to a physical, non-transitory, computer-readable medium, such as a tangible physical storage medium (e.g., an optical disc, a solid-state drive, etc.), which can be provided to a computer server or computing device to be loaded or executed various programs.

[0082] In the context of the disclosed method, the pipeline receives input for post-processing, which may include video data objects and target audio data objects. The system is configured to generate new output video data objects that effectively replace certain regions (such as regions within the mouth area).

[0083] This paper proposes variations of the computing architecture. For example, in an example embodiment, a single U-net is utilized, which has demonstrated strong performance in experimental analysis.

[0084] Variations of the masking method were also proposed, such as an improved mask that extends the masking region to the nose area instead of just below the nose, and the masking method was found to exhibit strong performance in experimental analysis. Attached Figure Description

[0085] The accompanying drawings illustrate embodiments by way of example. It should be clearly understood that the description and drawings are for illustrative purposes only and are intended to aid in understanding.

[0086] Embodiments will now be described by way of example only with reference to the accompanying drawings, in which:

[0087] Figure 1 This is a drawing illustrating an example lip-tapping system according to some embodiments.

[0088] Figure 2 This is an illustrative diagram illustrating a process for decomposing audio into phonemes and retrieving relevant view positions according to some embodiments.

[0089] Figure 3This is a block diagram of an unwrapping process according to some embodiments, in which an image is encoded into unwrapped code containing all image information.

[0090] Figure 4 This is a block diagram of a lip-syncing process according to some embodiments, in which facial expression codes (view positions) are extracted from audio and added to the code of the input frame to obtain an output frame synchronized with the audio segment.

[0091] Figure 5 This is a block diagram illustrating the training process of an untangled network according to some embodiments, in which the loss is defined on the latent code and on images with correct poses and expressions from a database.

[0092] Figure 6 This is an illustrative diagram illustrating a method for data synthesis under different poses and facial expressions (view positions) according to some embodiments.

[0093] Figure 7 It is a flowchart depicting the preprocessing of input video and audio.

[0094] Figure 8 It is a description of Figure 4 The flowchart shown is a performance diagram of the Lip Dubber system.

[0095] Figure 9 This is a schematic block diagram of a computing system suitable for use in video generation, according to some embodiments.

[0096] Figure 10 This is a schematic block diagram of a computer system according to some embodiments.

[0097] Figure 11 It is the first audio signal w a A visual representation comparing the spectrogram segments of the first audio signal with the spectrogram units of the second audio signal.

[0098] Figure 12 This is a flowchart illustrating the process of lip-syncing.

[0099] Figure 13A A machine learning topology diagram of an example voice-to-lip network according to some embodiments is shown.

[0100] Figure 13B An example sequence sampler according to some embodiments is shown.

[0101] Figure 14 This illustrates a machine learning topology of a voice-to-lip model according to some embodiments, the model being configured to extract lip keypoints, audio, and identity templates from reference videos corresponding to an individual.

[0102] Figure 15A This is an example lip-to-image network based on some embodiments.

[0103] Figure 15B This is an example lip-to-image network based on some embodiments.

[0104] Figure 16 This is an explanatory diagram showing two images defining the repair area.

[0105] Figure 17 This is an example flowchart illustrating face generation according to some embodiments.

[0106] Figure 18A and Figure 18B These are example method flows based on some embodiments. Figure 18A Extending to Figure 18B .

[0107] Figure 19 This is an example block diagram of components for a system used to perform lip-syncing, based on some examples.

[0108] Figure 20 An example computational process flow is shown that can be used as part of a processing pipeline in a real-world commercial implementation. Detailed Implementation

[0109] The amount of content available on television is increasing rapidly. Foreign films are becoming more popular in English-speaking countries, and international streaming platforms are making it easier for non-English speakers to access English-language content.

[0110] To better appeal to audiences who speak languages ​​different from the film in question, it is desirable to translate the film script and then dub it. However, standalone audio dubbing does not match the speaker's lip movements and can lead to inconsistencies in timing. Therefore, it is necessary to manipulate lip movements to match the dubbed film in any given language.

[0111] Therefore, there is a clear and growing need for systems and methods that, given a video V and an audio A representing a language L, can manipulate V to obtain a V' based on the audio A' representing language L', such that the lips in V' match the audio A'. For example, the audio A can be in English, and the audio A' can be in French.

[0112] This presents a challenging technical problem because videos are typically high-quality and high-resolution, such as 4K or higher, and subtle mismatches or slight noise can produce noticeable artifacts that should ideally be corrected and removed. As described in this paper, a solution is proposed to provide a system specifically configured to generate improved video data objects V' with modified regions (e.g., a mouth region covering the lips and surrounding area).

[0113] In a variant, the technical solution further includes a viewpoint synthesis step for synthesizing viewpoints that can be used to generate V' but do not exist in the original V (e.g., the original audio language does not have actors making specific lip or mouth expressions for the target viewpoints); and an unwrapping step that can be used to identify the control parameters required to generate V' from a time-coded set of input viewpoints (e.g., corresponding to A', in the form of an audio track of the target language) sent to the generator.

[0114] Figure 1 This is a drawing illustrating an example lip-tapping system 100 according to some embodiments.

[0115] System 100 includes an input 102 having a video V having audio A and audio A' representing language L, wherein audio A' is a translated audio A representing language L'. Output 104 is a video V' having audio A' representing language L', the video being arranged such that the frames of video V' match the frames of their respective audio A's.

[0116] In some embodiments, in addition to the audio A' of language L', the video V also includes frames F and the audio A of language L. System 100 manipulates frames F such that each frame I∈F is manipulated to obtain I'∈F' that matches the audio A'.

[0117] To match frames with audio segments, a deep neural network can be implemented, which receives a frame I∈F and its corresponding spectrogram unit s∈A', and generates a frame I' that matches s.

[0118] Figure 2 This is an illustration of a process 200 for decomposing audio into phonemes and retrieving relevant view positions according to some embodiments.

[0119] In phonology and linguistics, there are phonemes and foves. A phoneme is a unit of pronunciation that distinguishes one word from another in a particular language. For example, in most English dialects, pronunciation patterns... (sin) and (sing) are two different words that can be distinguished by replacing one phoneme / n / with the other phoneme / η / .

[0120] A phéme is, for example, any of several speech sounds that appear the same when lip-read. It should be noted that phémes do not have a one-to-one correspondence with phonemes. For a given audio track, phonemes and phémes can be time-encoded as they appear on the screen or audio, and this process can be performed automatically or manually. Therefore, A' can be represented as a data object that may contain a time-series encoded set of phonemes or phémes. For phoneme representation, in some embodiments, it can be converted to a phéme representation (if possible) via a lookup transformation. In another embodiment, the phoneme / phéme connection can be obtained by training a machine learning model iteratively through a supervised training dataset with speech transcriptions and corresponding frames as value pairs.

[0121] Typically, a single photic position can correspond to multiple phonemes because several different phonemes appear in the same way when produced on the face or lips. For example, words like pet, bell, and men are difficult for lip-readers to distinguish because they all look like / pet / . Or phrases like "elephant juice" look exactly the same as "I love you" when lip-read.

[0122] As an instance of a set of phonemes encoded in time series, and also as Figure 2 As shown, for A', a timestamped list of phonemes that marks the entire sequence can be generated based on the detected phonemes.

[0123] A time-series encoded set of viewpoints represents a keypoint. For example, for each frame in a source video (depending on the frame rate), a set of keypoints is retrieved or generated, indicating a new viewpoint that matches that frame. If there are 600 frames in the source video, there could be 600 sets of keypoints. In this case, the time series or timestamps can contain frame correspondences.

[0124] Phoneme temporal coding (the set of phonemes used to generate time-series codes) can be viewed as operating in a "continuous" time space (although the audio is still sampled). See-bit temporal coding (the set of see-bits used to generate time-series codes), on the other hand, is encoded into a discrete frame space.

[0125] A phoneme-to-viewpoint (P2V) codebook can be used to classify various phonemes and their corresponding viewpoints. For example, a P2V codebook can be a data structure representing a lookup table used to provide phoneme classifications with corresponding viewpoints. Classification is not always 1:1, because many phonemes can have the same viewpoint, or similarly, contextual cues may change the viewpoint associated with a particular phoneme. By untangling viewpoints with other features of the image, other facial features (e.g., anger) can be preserved.

[0126] Starting with audio signal A and related text, audio A is decomposed into segments s. i To find the corresponding phoneme p i According to p i Determine or extract the corresponding view position v i If the desired viewpoint and pose are available in the input video (see...). Figure 11 If the desired viewpoint is obtained from the original input, it can be retrieved; otherwise, it may need to be generated using the proposed unwrapping model as described herein. In some embodiments, the desired viewpoint can also be obtained from a library associated with a specific actor or character in other speaking roles in other videos.

[0127] View positions are added to a database of view positions that can be pre-synthesized, as will be described further below.

[0128] Ideally, any lip movement can be constructed by combining images representing these viewpoints.

[0129] However, such images depicting a specific viewpoint vary in terms of pose, lighting conditions, and other factors. Therefore, a mechanism can be applied to manipulate these frames to match them with specific viewpoints (expressions), poses, lighting conditions, and other different factors.

[0130] In some embodiments, the process includes classifying view positions or learning code for each image. Then, by replacing one piece of code and changing others, the machine learning model architecture ideally alters only one aspect of the image (e.g., the relevant mouth area).

[0131] The code might be a vector of length N. Depending on the machine learning model architecture, which may include generator networks such as StyleGAN, the length N and how the code is determined can vary.

[0132] For example, learning code examples Figure 3 As shown in w+, machine learning model architectures can learn code by finding some code that produces the same image when fed to the generator network.

[0133] The machine learning model architecture is then trained to find the necessary modifications to the code, thereby generating the desired viewpoint while preserving all other characteristics of the image. When modifying the image, the code for the "open mouth" shape of a person in the image should not cause the hair to turn red.

[0134] As a non-limiting example of process 200, audio with text can be received, and phonemes can be extracted from the received audio. Appropriate viewpoints can then be assigned to these identified phonemes, which can be done by looking up the corresponding viewpoints using a suitable P2V codebook.

[0135] Each frame I∈F consists of an expression e containing the geometry (i.e., view position) and texture of the lips and mouth, an identification string or number (ID) that distinguishes one individual from another, and a pose p that specifies the facial orientation.

[0136] In voice-over applications, only the relevant facial expressions can be modified based on the spectrogram s, while the pose p and everything else (the residue r) can remain unchanged. Therefore, the core neural network learns to unwrap e, r, and p from I.

[0137] Figure 3 This is a block diagram of unwrapping 300 according to some embodiments, in which the image is encoded into unwrapped code that retains all image information.

[0138] Untangling is a technique that decomposes or untangles features into narrowly categorized variables and encodes them as separate dimensions. The goal of untangling is to simulate the rapid intuitive processes of the human brain using both "high-dimensional" and "low-dimensional" reasoning.

[0139] In the illustrated example embodiment, unwrapping 300, image frames 310, and 320 are processed by multiple encoders 330 into three unwrapped codes, which represent pose, expression (view position), and residues containing all image information. For training, identities and pairs of images with the same pose, identity, or ID should be preserved. The pairs of data used for unwrapping can be encapsulated or represented in different forms (e.g., vectors, integers, 2D / 3D points, etc.). In some embodiments, the method includes intentionally overfitting the input video to achieve an improved result.

[0140] The described non-restricted neural network uses three encoders to dewrap the expression e and pose p with other image features, including ID, background, lighting, and other image features. The codes for these image features are integrated into code w via multilayer perceptron (MLP) networks 340a and 340b. + In 350a and 350b. w + Images 350a and 350b can be passed to pre-trained generators 360a and 360b, such as StyleGAN, to generate new images I'370 and 380.

[0141] MLP networks 340a and 340b are a type of neural network that consists of neurons in one or more layers. Data is fed into the input layer, and then possibly into one or more hidden layers that provide a level of abstraction, before predictions are made at the output layer, or "visible layer".

[0142] Encoder 330 and MLP networks 340a and 340b can be trained on identity tasks, meaning I and I' are identical, or they can be trained on paired datasets of I and I' that are different in one or two features (e.g., ID, pose, or expression). For lip-syncing purposes, expressions can be obtained from a viewpoint database. During output video generation, I' can be a complete image or a selected mouth region, and either can be inserted to generate a replacement video frame. Inserting only the mouth region may be faster and computationally cheaper, but this can have issues with bounding box regions and inconsistencies with other aspects of the video not within the replacement region.

[0143] The training will be described in further detail below.

[0144] Figure 4 This is a block diagram of a lip-syncing process 400 according to some embodiments, in which facial expression codes (view positions) are extracted from audio and added to the code of an input frame to obtain an output frame synchronized with an audio segment.

[0145] The code for the input frame here can be generated using a latent space inversion (or encoding) process.

[0146] Modifications to vectors or codes allow for semantic modifications when the image is passed back through the generator. For example, moving along the direction of "age," represented by a vector in the latent space, will make a person in the generated image appear older.

[0147] Image frame I 410 is processed by multiple encoders 430a, 430b, 430c into three unwrapped codes, which represent the pose p, expression (view position) e, and residue r containing all image information.

[0148] The non-limiting embodiment process 400 in this document implements three encoders 430a, 430b, and 430c, which are used to dewrap the expression e and pose p from other image features, including ID, background, lighting, and other image features. The codes for these image features are integrated into code w via a multilayer perceptron (MLP) network 440. + 450 in. w + 450 can be passed to a pre-trained generator 460 such as StyleGAN to generate new images I'470.

[0149] In some embodiments, a separate audio track for each individual character is obtained (or extracted from a combined audio track). For example, heads and faces can be identified by using a machine learning model to detect faces and build normalized bounding boxes. Distal and proximal heads may be approached differently, as proximal heads may have a larger number of pixels or image regions to modify, while more distal heads have a smaller number of pixels or image regions to modify.

[0150] In order to perform the lip-syncing process 400 shown in the example embodiment, facial expression codes (view positions) are extracted from the audio and added to the code of frame I to obtain frame I' synchronized with the audio segment.

[0151] In order to perform lip-syncing, audio A' undergoes a positional recognition process, so that for each spectrogram segment s i Locate the view position. The system can be configured to map audio to phonemes, and then map the phonemes to view positions.

[0152] For example, 19 view positions can be considered and indexed by a single unique integer (1-19). The spectrogram si can then be passed to another encoder or a separate module (such as a phoneme-to-view position module) to obtain the data from si. i The product generated in the middle is called e s The expression / viewpoint code. The input video may or may not have the same viewpoint as I. If V already has the same viewpoint and pose, it can be easily retrieved (see [link]). Figure 11 If not, I is first encoded into three potential codes containing eS, r, and p. Then, instead, e is encoded into... s r and p are passed to the decoder to generate a new frame I', which retains the ID, pose, etc., while also being associated with the expression e from the audio. s match.

[0153] In some embodiments, the mouth region may be extracted only from I' and inserted into I, and image reconciliation may be performed to generate a smooth result.

[0154] It should be noted that the potential code can be of any size or form, including hot code, a single integer value, or a floating-point vector of any size.

[0155] In other embodiments, it is preferable to copy the entire I' or simply create the lip shape and insert it back into I.

[0156] Additionally, if the correct pose and expression are already available in the input video V, suitable frames can be simply retrieved from video V. If no such frames exist, new frames can be generated using the process discussed. The example generator described may use StyleGAN or a variant thereof.

[0157] In some embodiments, an additional feedback process is considered using a lip-reading engine that automatically generates output video / text and then feeds it back to the system to compare it with the input to ensure that the output video is authentic.

[0158] Figure 5 This is a block diagram of an untangled network training process 500 according to some embodiments, in which the loss is defined on the latent code and on images with correct poses and expressions from a database.

[0159] For the training process 500, which may be the first untangled network, according to some embodiments, I and I' have been paired and have been improved in terms of realism by pSp.

[0160] Pixel2style2pixel (pSp) is an image-to-image translation framework. The pSp framework provides a fast and accurate solution for encoding real-world images into the latent space of a pre-trained StyleGAN generator. Furthermore, the pSp framework can be used to solve various image-to-image translation tasks, such as multimodal conditional image synthesis, face frontalization, inpainting, and super-resolution.

[0161] In some embodiments, pSp can be used to map images created in a compositing environment with different viewpoints, poses, and textures to images that appear realistic.

[0162] To do this, the synthesized image can be fed into pSp and code w0 can be generated.

[0163] In another embodiment, sampling can also be performed in a real domain called w1. By mixing the top entries of w0 with the bottom entries of w1, the facial expressions (e.g., view positions) and poses of the synthetic image captured in w0 can be preserved, and a realistic image that looks similar to a real image with code w1 can be produced. By sampling different images and generating various w1s, some embodiments can produce a large number of labeled realistic images in certain poses and view positions indicated by the synthetic data. This labeled realistic data can be used to learn untangling.

[0164] The loss L1 (e.g., |xi-xj|) can be defined based on the results and the ground truth. I' can also be fed back to the video encoder to obtain r', p', and e', and compared with the input code. For this purpose, r', p', and e' can be defined relative to i, p, and e. s L1 loss. To ensure the new lips are effective, the database should be searched for values ​​related to r, p, and e. s The closest image.

[0165] Figure 6This is an illustrative diagram of data synthesis 600 under different postures and expressions (view positions) according to some embodiments.

[0166] To understand the different characteristics of frame wrapping, a relevant dataset is needed. To generate such a dataset, data can be synthesized using different identities rendered under different poses and expressions. These expressions contain all available viewpoints required to produce valid lip-syncing.

[0167] The uncanny valley in aesthetics is a hypothetical relationship between the degree of resemblance of an object to a human and the emotional response to that object. This hypothesis suggests that humanoid objects that are not entirely similar to real humans evoke a sense of familiar grotesqueness and revulsion in the observer. The "valley" refers to the sharp decrease in human observers' affinity for replicas, which otherwise increases with the replica's human-like resemblance. For example, certain lifelike robotic dolls that appear almost human risk triggering a chilling, grotesque feeling in the viewer.

[0168] To overcome the uncanny valley and produce more realistic images, synthetic datasets will be fed into pSp to generate natural images with different IDs.

[0169] Therefore, according to some embodiments, the described system learns to unwrap facial expressions (view position and lip shape) from other characteristics such as pose, lighting, and overall texture. Therefore, data is needed to learn how to unwrap these characteristics.

[0170] Another embodiment realistically synthesizes missing viewpoints. This is necessary when the correct viewpoints are unavailable in the input video. This can be particularly useful when the input video is short. According to some embodiments, this is done by utilizing the system to generate synthetic data with different poses and IDs, and additional steps described in the data synthesis 600 above can be performed to make it more realistic.

[0171] Figure 7 This is flowchart 700, which depicts the preprocessing of input video and audio.

[0172] In some embodiments, lip-syncing can consist of two parts. Flowchart 700 depicts the first part, preprocessing. In preprocessing, the viewpoints of input 102 are found and added to a database. Audio A' is processed to identify the viewpoint codes of its audio segments.

[0173] The second part of the embodiment relates to lip-syncing.

[0174] Figure 8 It describes as Figure 4 The flowchart shown is a block diagram 800 illustrating the performance of the lip-syncing voice actor. Based on the view code of audio A', Figure 4The lip-syncing depicted in the video can be used to modify frames of video V.

[0175] Figure 9 This is a schematic block diagram of a computing system 900 suitable for use in video generation, according to some embodiments.

[0176] The system can be implemented using a computer processor or a set of distributed computing resources provided by a system for generating special effects or modifying video input. For example, the system could be a server specifically configured to generate lip-sync video output, wherein it receives input video and executes translation subroutines or procedures to modify the input video to generate a new output video.

[0177] As described above, System 900 is a machine learning engine-based system comprising various maintained machine learning models that are iteratively updated and / or trained, having interconnect weights and filters that are tuned to optimize for specific features (e.g., through a defined loss function). Multiple machine learning models can be used together in a coordinated manner; for example, as described herein, a specific set of machine learning models can be used first to unwrap specific parameters, thereby ultimately controlling the video generator illusion network.

[0178] Figure 9 The computing elements shown are examples and can vary, and more, different, or fewer elements can be provided. Furthermore, computing elements can be implemented as computing modules, engines, code routines, logic gate arrays, etc., and in some embodiments, system 900 is a dedicated machine suitable for video generation (e.g., a rack-mounted device coupled to a computing data center via a message bus).

[0179] This system can be used, for example, to computationally automate previous manual lip-syncing / redrawing exercises, and the problem that is associated with previous methods is lip-syncing, where the male / female actor speaking the target language must match syllables with the original lip movements (resulting in timing or script inconsistencies in the target language) or the on-screen lip movements do not correspond correctly to the audio in the target language (mouth movement without speech, or no movement but the character is speaking).

[0180] The input data set is obtained at 902, for example, as a video feed provided by a studio or content creator, and can be provided, for example, as streaming video, or as video data objects (e.g., .avi, .mp4, .mpeg). The video feed can have associated audio tracks that can be provided individually or together. The audio tracks can be broken down by different audio sources (e.g., different feeds from different screen characters in a recording studio).

[0181] Target audio or script may be provided, but in some embodiments, it is not provided, and the target audio or script may be synthesized using machine learning or other generative methods. For example, instead of having a new voice actor speak in a new language, the method obtains a machine translation and automatically uses the generated voice.

[0182] The viewpoint extraction engine 904 is adapted to identify necessary viewpoints and their associated time codes from target audio or script. In some instances, these viewpoints can be extracted from phonemes if provided, or from the video using a machine learning engine. Viewpoints can be mapped to a list of all viewpoints and stored as tuples (e.g., viewpoint 14, t = 0.05–0.07 seconds, character Alice; viewpoint 13, t = 0.04–0.08 seconds, character Bob).

[0183] The viewpoint compositing engine 906 is configured to compare necessary viewpoints with a known set of viewpoints from the original video data object, and to perform compositing on any missing viewpoints from the original video data object as needed. This compositing can include obtaining viewpoints from other works by the same actor, generating all new mouth movements based on "feature faces," and so on.

[0184] The view position untangling engine 908 is a set of machine learning models that are individually tuned to decompose or separate mouth movement associated with various view positions when controlling the machine learning generator network 912, which is then used to generate control parameters using the control parameter generator engine 910.

[0185] Then, the machine learning generator network 912 (e.g., StyleGAN or another network) is operated to generate new frame objects whenever a person or character speaks or based on a viewpoint-time code of the target viewpoint. Frame objects can be partial or full frames, and in some embodiments, they are inserted into V to arrive at V'. In some embodiments, V' is not inserted into V, but is simply generated entirely by the machine learning generator network 912.

[0186] Output data set 914 is provided to downstream computing mechanisms for downstream processing, storage, or display. For example, the system can be used to generate ongoing events in multiple different languages ​​(e.g., news broadcasts), translations of film / TV program / animation outputs to a certain extent concurrently, etc. In another embodiment, output data set 914 is used to re-dub characters in the same language (e.g., when the original audio is unavailable for some reason or simply does not meet expectations). The system can also be used to modify accents (e.g., to correct different English accents, Chinese accents, etc.).

[0187] For example, the output dataset 914 can be used for post-processing of animation, where instead of drawing the initial face or mouth in the original video, the output video is generated directly based on a time-synchronized set of viewpoints, and, for example, the mouth or facial regions are drawn directly as part of the rendering step. This reduces the effort required to prepare the initial video for input.

[0188] In yet another instance, a system is provided that provides positional data and generates video without the original input video, and generates a completely “illusionary” video based on a set of instructions or storyboard data objects, the video corresponding to the correct mouth shape and mouth movement of the target audio track.

[0189] Figure 10 This is an example computing system according to some embodiments. Under software control, the computing device 1000 can... Figure 9 The schematic box shown controls the architecture of the machine learning model.

[0190] As shown in the figure, the computing device 1000 includes one or more processors 1002, a memory 1004, a network controller 1006, and one or more I / O interfaces 1008 that communicate via a message bus.

[0191] The processor 1002 can be one or more Intel x86, Intel x64, AMD x86-64, PowerPC, ARM processors, etc.

[0192] Memory 1004 may include random access memory, read-only memory, or persistent storage devices such as hard disks, solid-state drives, etc. Read-only memory or persistent storage devices are computer-readable media. Computer-readable media (e.g., non-transitory computer-readable media) can be organized using a file system and controlled and managed by the operating system that governs the overall operation of the computing device.

[0193] The network controller 1006 serves as a communication device to interconnect the computing device with one or more computer networks (e.g., a local area network (LAN) or the Internet).

[0194] One or more I / O interfaces 1008 can be used to interconnect a computing device with peripheral devices (e.g., a keyboard, mouse, video display, etc.). Such peripheral devices may include the display of device 120. Optionally, network controller 1006 can be accessed through one or more I / O interfaces.

[0195] Software instructions are executed by processor 1002 from a computer-readable medium. For example, software may be loaded into random access memory from persistent storage of memory 1004 or from one or more devices via I / O interface 1008 for execution by one or more processors 1002. As another example, software may be loaded and executed directly from read-only memory by one or more processors 1002.

[0196] The example software components and data stored in the memory 1004 of the computing device 1000 may include software that performs machine learning to generate illusion video data output, as disclosed herein, and operating system (OS) software that allows communication and application operation in relation to the computing device 1000.

[0197] According to one embodiment of this application, video V (image frame F + sound w) a The language L (e.g., English) is given, as well as the sound w. b The language L' (e.g., French) is given. Figure 11-17 The various processes that can be used to replace the lip shape in video V based on the sound of language L' are demonstrated.

[0198] Figure 11 It is the first audio signal w a A visual representation 1100 comparing the spectrogram segment of the first audio signal with the spectrogram unit of the second audio signal. F 1102 shows a set of example timestamped frames.

[0199] First audio w a Signal 1104 can be the audio signal of language L', specifically the audio signal A'. The second audio signal w b 1106 can be the audio signal of the audio A of the language L in the input video V. The second audio signal w b Each segment in the spectrogram can have a known view position and pose that can be obtained from the input video V. The audio signal w a Can be used with audio signal w b Alignment to identify the first audio signal w a The same second audio signal w b The spectrogram segment.

[0200] audio signal w a Can be used with audio signal w b Alignment to determine the second audio signal w b The corresponding view position of the spectrogram segment. As shown in the illustration, the first audio signal w a Some spectrogram segments can be associated with the second audio signal w b Some spectrogram segments are the same ( Figure 11(as shown in the green frame). For each common spectrogram segment, the known view position and pose of the spectrogram segment corresponding to the first audio signal can be retrieved, and used to determine the view position and pose of the spectrogram unit of the second audio signal.

[0201] In some embodiments, frames of video V that match these common spectrogram units can be copied from video V and used to generate video V'. For the remaining spectral bands that do not share commonalities, [the following can be used]. Figure 12 The process is shown.

[0202] This is an optional step that can be used to bypass certain similar frames to reduce overall computation time. For example, the sample output at this stage can be identified as segments that require frame generation (e.g., identified by time frames or duration). For example, these segments could represent all frames between two timestamps. For instance, there might be a video where there is speech between two people from t=5 seconds to t=6 seconds. However, similar frames have been identified for some speech from t=5.00 seconds to t=5.3 seconds and from t=5.5 seconds to t=6.0 seconds. Therefore, the frames from t=5.3 seconds to t=5.5 seconds can be inserted into the processing pipeline to generate frame segments representing the replaced mouth portions of these frames. Each frame in the frame can be processed together using two trained networks to replace its mouth portion, as described below.

[0203] Figure 12 This is a block diagram 1200 used for the process of lip-syncing.

[0204] The process can be used in situations where frames cannot be simply copied from the input video V, such as regarding Figure 11 As explained, the process may include a sound-to-lips step and a lip-to-image step. As shown in Figure 18, the described lip-syncing process can be performed using system 1800. System 1800 may be part of system 900 and may include a sound-to-lips network and a lip-to-image network. The sound-to-lips network may be a transformer neural network.

[0205] Transducer neural networks are a type of neural network that learns context and thus meaning by tracking relationships in sequential data. Voice-to-lip networks can be used to personalize the geometry of the lips based on the speaker (through fine-tuning). The voice-to-lip step may involve receiving the geometry of the lips and animate them based on the sound or audio signal.

[0206] The lip-to-image step may involve receiving the personalized geometry of the lips (based on the audio) and each frame that needs to be dubbed. As will be described in further detail below, each frame to be dubbed can first be analyzed to extract the existing lip shape for the purpose of masking the lips and chin.

[0207] As will be described in further detail below, the lip-to-image step can then be tasked with “filling” this masked region corresponding to a given lip shape. The mask is a crucial step because without it, the network cannot learn anything and can only copy from the input frame.

[0208] like Figure 12 As shown, there are pre-training step 1202 and inference step 1204.

[0209] During the pre-training step 1202, both the voice-to-lips 1206 and lip-to-image 1208 models are trained, for example, using the individual identities or identity+shift pairs, allowing the model interconnections and their weights to be refined over a set of training iterations. Depending on what is available in the training set, a different set of faces can be trained.

[0210] During reasoning step 1204, before reasoning about a particular individual, both the voice-to-lips 1206 and lip-to-image 1208 models of the particular individual can be fine-tuned.

[0211] sound to lips

[0212] Figure 13A An example sound-to-lip network 1300 according to some embodiments is shown. The sound-to-lip network can use a transducer-based architecture. The sound-to-lip network can be trained end-to-end to autoregressively synthesize lip (and chin) keypoints to match the input audio. As shown, the transducer model can include a transducer encoder that encodes the input audio into “tokens,” and a transducer decoder that processes the audio tokens and previous lip keypoints to synthesize a sequence of lip keypoints. The transducer encoder is compatible with Wav2Vec2.0. TM The design matches the data and can be initialized using its pre-trained weights. Wav2Vec2.0 is a self-supervised learning model for speech representation.

[0213] The Wav2Vec 2.0 model was trained on 53,000 hours of audio (CC by 4.0 licensed data), making it a powerful speech encoder. (Compared to FaceFormer) TMCompared to other models, this application focuses on the explicit generation of lips (rather than the entire face) and lip personalization for new identities not included in the training set. Faceformer TM It is a converter-based autoregressive model that encodes long-term audio context and autoregressively predicts a range of animated 3D facial meshes.

[0214] The voice-to-lips model aims to address three problems in existing methods.

[0215] Limitations on Data: The most common datasets for voice-to-lip models are the BIWI and VOCASET datasets. These datasets consist of audio clips from multiple identities along with extremely high-precision tracking meshes of the speakers (i.e., 3D models of the faces). The problem this presents is that fine-tuning the model is impossible due to the need for similarly high-quality meshes of the target identities.

[0216] Identity Templates: Additionally, since the BIWI and VOCASET datasets were created in a "clean" (impractical) setting, these datasets can provide a grid of identity templates for making predictions based on them. This again limits the ability to fine-tune new identities, as obtaining this grid is impractical.

[0217] Lip Style: Finally, FaceFormer learns each speaker's "style" through an embedding layer that takes one-hot embeddings typed from the identities of the lips and voices in the training set as input. This choice limits the model to predicting lips based on one of the identities in the training set. Using another individual's lips to make predictions could be problematic because the geometry of an individual's lips is unique.

[0218] As described in this paper, a voice-to-lip model can be trained to predict an individual's lip keypoints based on any video containing image frames capturing the individual speaking. The advantage of directly processing the video is that keypoints extracted for training purposes can be drawn from any video, enabling fine-tuning of the target shot.

[0219] The voice-to-lip model is configured to extract lip keypoints, audio, and an identity template from a reference video corresponding to an individual. The reference video is labeled with the individual's identity. The identity template can be a 3D mesh of the individual's lips. This data is then smoothed to reduce noise (removing high-frequency noise) before being used for training. In some embodiments, the voice-to-lip model can extract 40 keypoints from the lips and 21 keypoints describing the jawline, for a total of 61 keypoints. It should be understood that different numbers of facial keypoints (e.g., lip keypoints) can be extracted.

[0220] Figure 14 A block diagram 1400 of a voice-to-lip model is shown, the voice-to-lip model having a data creator model 1402, the data creator model being configured to extract lip key points 1404, audio 1406 and identity template 1408 from a reference video 1410 corresponding to an individual identity 1412.

[0221] The identity template can be a 3D mesh of an individual's lips. A set of synthetic lip or chip keypoint data tailored to the individual can be determined based on the deviation from a specific identity template 1408.

[0222] Identity template 1408 can be extracted in a variety of ways. For example, identity template 1412 can be based on a "still" pose image (in... Figure 14 The identity template 1408 is generated by averaging all keypoints extracted from a reference video corresponding to the individual. This idea of ​​a "still" pose image is closely related to the BIWI and VOCASET datasets, which provide similar identity template grids. However, this approach is limited because "still" pose images may not be usable for new identities. In this invention, the individual's identity template 1408 is generated from the average of all keypoints extracted from a reference video corresponding to the individual. Providing a single identity created from the average of all extracted lip values ​​not only performs better but also solves the problem of deciding which template to use for δ prediction.

[0223] Finally, due to the importance of lip style (personalization) to the generation process, this method attempts to eliminate the reliance on the "one-hot" identity specification present in FaceFormer. Instead of restricting the model to "one-hot" lip recognition based on identity styles in the training set, this invention seeks to learn the speaker's "style" from a given lip sequence of an individual. For example, the model can sample keypoint sequences from another dataset instance for a given identity. This keypoint sequence can then be used to inform the speaker's style. The idea is that by swapping the sampled sequences of each sample (but ensuring they come from the same identity), the "style embedding" layer will be able to adapt to new identities at test time.

[0224] Figure 13B An example sequence sampler 1350 according to some embodiments is shown. The sequence sampler may include multiple mouth shapes based on identity 1302, frame 1304, and video 1306.

[0225] By extracting lip keypoints and sounds from the original video and specifically adjusting the "style encoder" for the new target identity, the voice-to-lip model mapping is fine-tuned for the new identity. Once fine-tuned, the voice-to-lip model can generate lips from any audio in the style of the target identity.

[0226] Lips to Image Network

[0227] Figure 15A This is an example lip-to-image network 1500 according to some embodiments.

[0228] As depicted, the lip-to-image network 1500 comprises a first stage and a second stage. In the first stage of the network, a mask frame 1502 and keypoint codes 1512 learned from the geometry of the lips and jaw (see explanation below) are received to produce a coarse estimate or intermediate result 1504 of the reconstructed frame. The reconstructed frame may omit some details. In the second stage of the network, appearance codes from the previous stage and the intermediate result 1504 are received to produce a detailed reconstruction as an output sequence 1506. The detailed reconstruction may include details previously omitted in the intermediate result 1504.

[0229] The lip-to-image network 1500 may include a transformer encoder to encode the lip geometry of target lips and jaw keypoints. This encoding of the target geometry is referred to as "keypoint code" 1512. As depicted, keypoint code 1512 can be passed to both the individual codebook 1508 and the first stage of the network via an adaptive group-wise normalization layer. Note that appearance code can be learned based on ID. To obtain the appearance code, a personalized codebook 1508 can be learned for each identity. Then, a set of coefficients or weights 1510 can be estimated based on the keypoint code, which is multiplied by the feature vector of the codebook to produce the final appearance code.

[0230] For these two stages, a UNET network similar to the DDPM structure can be used.

[0231] To ensure believable lip geometry and texture, Network 1500 can be trained first on an initial dataset of individual speakers and then fine-tuned to fit the video of a single actor speaking. This fine-tuning process biases Network 1500 towards generating lip geometry and texture specific to the target actor being dubbed. Note that individual codebooks can be learned first on the entire dataset and then fine-tuned for their identity.

[0232] In some cases, the lips in the input frame can be closed, and the lips in the output frame can be open. In other cases, the lips in the input frame are open, and the lips in the output frame are closed. For example... Figure 16 As shown, system implementation 1600 addresses these situations by masking the input frame 1602 with a masking region 1606 based on the maximum area covered by the jaw, thereby reducing potential texture artifacts in the detailed reconstruction frame 1604. The masked frame can define a repair region 1608 for generating at least one of the coarse reconstruction frames (i.e., intermediate results) and the detailed reconstruction frames. This is crucial, otherwise, artifacts in the texture may appear, such as a double chin.

[0233] The Lips to Image Network 1500 can utilize a variety of losses. Several example losses are described below, for example, using first, second, third, and / or fourth losses that can be used together in various combinations to construct the overall loss function for optimization.

[0234] The first loss could be a mean squared error loss, used to measure the squared difference of pixel values ​​between the ground truth image of Network 1500 and the output image. The second loss could be a learned perceptual image patch similarity (LPIPS) loss, which measures the difference between patches in the ground truth image relative to the output image of Network 1500. The third loss could be a height-width loss, which measures the difference in lip opening between the ground truth image and the network output. The neural network can be used as a differentiable module to detect keypoints on the output lips as well as the ground truth image, and compare the differences in lip keypoints (i.e., the fourth loss). Finally, a lip synchronization expert discriminator is used to correct the synchronization between the audio and the output.

[0235] The lip-to-image network works directly on generator networks such as, but not limited to, StyleGAN. The method learns a set of codes representing view positions, and then, for each lip shape, the network generates a set of coefficients; multiplying these coefficient sets by the codes produces any lip shape.

[0236] This expressiveness allows movement along a certain direction to cause localized, meaningful edits to the face for a given point in the potential space representing the face. For example, movement in one direction might turn black hair blonde, and movement in another direction might make the lips appear to be smiling.

[0237] The problem addressed by this method is finding orientations in the latent space of a generator (e.g., StyleGAN) that represent different lip movements when a person is speaking. The applicants address this problem by recognizing that human lip movements can be broadly categorized into a finite number of groups, which, if learned, can be combined to create any arbitrary lip shape.

[0238] In some embodiments, the system may include a machine learning architecture with only a single UNET network. Figure 15B This is another example of a lip-to-image network 1550 according to some embodiments.

[0239] As depicted, the lip-to-image network 1550 contains only a single UNET network. The UNET network model receives and processes the mask frame 1502, keypoint codes 1512 learned from the lip and jaw geometry, and optionally appearance codes, to produce the final reconstructed frame 1506, thereby skipping... Figure 15AIntermediate results in the process. In some embodiments, appearance codes are not used to generate output sequence 1506.

[0240] The lip-to-image network 1550 may include a transformer encoder to encode the lip geometry of target lip and jaw keypoints. This encoding of the target geometry is referred to as "keypoint code" 1512. As depicted, the keypoint code 1512 can be passed to both the individual codebook 1508 and the network via an adaptive grouping normalization layer. Note that an appearance code can be learned based on an ID. To obtain the appearance code, a personalized codebook 1508 can be learned for each identity. Then, a set of coefficients or weights 1510 can be estimated based on the keypoint code, which is multiplied by the feature vector of the codebook to produce the final appearance code.

[0241] Figure 17 An example architecture 1700 of the model is shown, which illustrates multiple steps in face generation.

[0242] In the first step, the system modifies the lip shape of each frame of a given video to a canonical lip shape and encodes the image into the StyleGAN latent space using E4E. Lip shape normalization can be done in several ways. One approach is to mask the lower region of the face, similar to the UNet method, and train the encoder from scratch to learn the canonical lip shape. Another approach is to apply the Gangealing process 1702 to each frame, take the average of the frames in the condensation space, and paste the lower portion of the average image back into each frame. Compared to the masking method, this approach has the advantage of avoiding training the encoder from scratch using a pre-trained E4E encoder and avoiding the loss of details in the lower facial region due to masking.

[0243] In the second step, the system is adapted to learn editing directions that transform a canonical lip shape into an arbitrary lip shape represented by a set of lip keypoints 1704. This is accomplished by representing different lip movements in the StyleGAN space using a linear combination of a set of learnable orthogonal directions 1708. Each of these directions should represent a change from the canonical lip shape to the view position, and combinations of these view positions can be used to generate any arbitrary lip shape. The applicants view the problem of learning these directions as a reconstruction problem, where the network directly optimizes the directions by learning to transform the canonical lip shape of each frame into the correct lip shape during training.

[0244] More precisely, the applicants first extract keypoints 1704 from the face in a given frame and pass them through an MLP to determine the coefficients of the linear combination. The system then orthogonals the orientations using the Gram-Schmidt method and computes the linear combination. Finally, the system adds the combination to the latent code specified by the E4E encoder.

[0245] In the final step, the system passes the resulting latent code from the previous step to the pre-trained StyleGAN generator and outputs image 1710. The training process is supervised by the L2 and LPIPS losses between the generator output and a given frame.

[0246] In order to perform lip-syncing on a given video, in this embodiment, the system can obtain a stream of lip key points from the Voice2Lip network and pass it into the frame, instead of extracting lip key points from the frame.

[0247] Figure 18A and Figure 18B This is a flowchart of a process for mapping across two pages according to some embodiments (including sub-processes 1800A and 1800B), which illustrates a method for generating output video using machine learning methods.

[0248] exist Figure 18A The process 1800A is shown to demonstrate how to train and fine-tune an autoregressive model for inferring lip shape based on audio.

[0249] Initial LipFormer process and steps

[0250] Subprocess 1800A begins with training data, and within this method, instances related to systems used for lip formation (e.g., LipFormer) are described. Training data for LipFormer can be video recordings in which a single speaker is speaking into a camera within the field of vision. This data can be collected by recording internal employees speaking predefined sentences targeting a series of visual positions (lip shapes).

[0251] Once this data is collected, the system can begin the lip-former preprocessing procedure. For each video in the dataset, the process may include:

[0252] 1. Detect faces and key points for each frame in the video.

[0253] 2. Use Procrustes analysis to project 2D pixel space keypoints into normalized 3D.

[0254] a. The canonical representation moves all keypoints to a common space, unaffected by the facial positions in the image.

[0255] 3. Extract audio from video

[0256] 4. Input audio, key points, and identity data into the dataset for training.

[0257] a. For simplicity, the identity marker is displayed on the video.

[0258] The machine learning model LipPuppet is trained to generate lip keypoints given only audio and identity. The system can train LipPuppet on a "global" data pool and then fine-tune the model in subprocess 1800B based on any new identities. Without fine-tuning, the global model can produce lip shapes that match any of the trained identities, but will not capture details of specific unseen identities.

[0259] LipPuppet can be used directly without fine-tuning, but lips will not capture the intricate details of each unique identity. If data is available for fine-tuning, the following process can be used to tailor LipPuppet to the identity of interest.

[0260] 1. Process the given identity shot based on "Data and Preprocessing".

[0261] 2. Load the global LipPuppet model

[0262] 3. Initialize the "style" embedding to be learned

[0263] 4. Optimize the "Style" embedding layer, and freeze (or unfreeze) other LipPuppet layers.

[0264] 5. Fine-tune the training data until convergence.

[0265] The goal of fine-tuning is to learn the "style" of any speaker not in the training set.

[0266] Once fine-tuned, the reasoning process can include:

[0267] 1. Load LipPuppet models for specific identities (or globally).

[0268] 2. Divide the audio segment into segments of length N milliseconds (LipPuppet has the maximum sequence length).

[0269] 3. Overlapping audio segments in K milliseconds

[0270] 4. Proceed each segment forward.

[0271] 5. Connect the generated lip key points and average the results in the overlapping area.

[0272] Lip keypoints can now be used with lip2face. Note that the "Dubbing Manager," which is not discussed here, can be configured to apply filtering to the lip keypoints before passing them to lip2face. This filtering helps with the transition between silences in the dubbing track and moments when the lip shape matches between the source and the dubbing.

[0273] The initial Lip2Face workflow and methodological steps are described, covering how the system trains and fine-tunes the model used to fill in lip textures given a lip shape and masked input frames. Except for not requiring audio, Lip2Face's data requirements are similar to LipFormer. Lip2Face may require the original frames along with their extracted keypoints.

[0274] 1. Detect facial features and 2D key points for each frame in the video.

[0275] 2. Crop and rotate the image to the face.

[0276] a. Rotation keeps the eyes in the same position.

[0277] 3. Generate a mask to cover the mouth area.

[0278] a. It can start from the tip of the nose downwards, or along the contour of the chin.

[0279] b. It may also include the nose area and extend to cover the mouth area.

[0280] 4. Describe the image, cropping, masking, and key points.

[0281] In some embodiments, masking from the tip of the nose downwards (i.e., excluding the nose) can lead to information smuggling during training, causing the machine learning model to overemphasize smile lines or cheeks in the input frames. Therefore, masks that include other regions, such as masks that include the nose region, can provide technically and significantly improved results.

[0282] The presence of certain facial expressions (such as smile lines) limits the flexibility of machine learning models in locating lips on a face. This is because smile lines are also taken into account (i.e., interpolated) when generating new lip shapes suggested by the neural network of the machine learning model. Therefore, during training, the network needs to balance the desired lip shape (suggested by lip geometry conditions) with the constraints imposed by the smile lines present in the input video.

[0283] For example, a person in an input video might have smile lines. In fact, a person cannot simultaneously make an "ooo" mouth shape while having smile lines. During training, the machine learning model might unintentionally receive cues about lip shape from information hidden within smile lines, leading to information smuggling. This can cause the model to overemphasize smile lines or cheeks during inference, resulting in inaccurate lip shape predictions. Therefore, masking one or more regions of the face that are highly correlated with lip shape improves the performance of the machine learning model.

[0284] In some embodiments, the input image is used only for texture, and the lip keypoints are used only for mouth shape.

[0285] Lip2Face is trained to “fill” a given masked input image using given lip keypoints from the frame and optionally for an identity. The system trains Lip2Face on a “global” data pool and can then fine-tune it for specific identities to better capture texture. In another embodiment, the model can be used without fine-tuning, but fine-tuning will improve results if data is available.

[0286] Fine-tuning

[0287] As mentioned above, Lip2Face can be used directly without fine-tuning, but the generated lip texture may not be of high quality. If the data is available, Lip2Face can be fine-tuned using the following process.

[0288] 1. Process the given identity shot based on "Data and Preprocessing".

[0289] 2. Load the global Lip2Face model

[0290] 3. Initialize the "Style" embedding

[0291] 4. Optimize "Style" embedding, freeze other lip2face layers (or do not freeze them).

[0292] 5. Fine-tune the training data until convergence.

[0293] 6. The goal of fine-tuning lip2face is to learn the “style” embedding that represents this new, unseen identity.

[0294] Lip2Face's inference process can use keypoints generated by LipPuppet. Lip2Face can also use keypoints extracted directly from video footage, which simplifies the process. The following procedure is used to create new dubbed frames from lip keypoints.

[0295] 1. Process the video to be dubbed based on the "Data and Preprocessing" steps.

[0296] 2. Load the driving lip keypoints generated by LipPuppet

[0297] 3. Use a source lip transformation from pixel to canonical space to align the driving keypoints with the extracted "source" lips.

[0298] 4. (Optional) Least-squares lip personalization of lip key points to match the source.

[0299] a. If LipPuppet tuning was not performed or was unsuccessful, then you can use

[0300] 5. Proceed forward through each mask source frame, replacing keypoints with loaded and aligned driving keypoints.

[0301] 6. Create a video based on the reasoning results and add voice-over audio as an audio track.

[0302] At the end of this process, the output is a dubbed video. The "Dubbing Manager" process can be used again here to replace unwanted frames (for example, frames that are being removed when a character is laughing or when the dubbing is silent compared to the original file).

[0303] Figure 19 This is an example schematic block diagram 1900 of components for performing lip-syncing, based on some examples. In schematic block diagram 1900, a set of computational processes is shown, including different machine learning models and program code execution blocks, which can be implemented as a modular computer program stored on a non-transitory computer-readable storage medium.

[0304] Figure 20 An example computational process flow 2000 is shown that can be used as part of a processing pipeline in a real-world commercial implementation. Figure 20 The figures illustrate steps that can be performed in parallel and in series according to the various embodiments described herein, such that computational input is received, the model is trained, and the trained model is deployed to automatically generate output.

[0305] The system can be implemented as a dedicated machine, such as a dedicated computing device that can operate as part of a computer server or as a computer server. For example, a rack-mounted device can be used for a specific purpose in a data center: receiving input video on a message bus as part of a processing pipeline to create output video. Dedicated machines can be used as part of a visual effects post-production computational approach, for example, editing after the initial material has been created. Editing can involve the integration of overlays or introduced computer graphics elements to replace parts of live-action footage or animation, and this editing can be computationally intensive.

[0306] A dedicated machine can be instructed according to a set of machine-interpretable instructions, which causes a processor to perform the steps of a computer-implemented method. The set of machine-interpretable instructions can be attached as an article of writing to a physical, non-transitory, computer-readable medium, such as a tangible physical storage medium (e.g., an optical disc, a solid-state drive, etc.), which can be provided to a computer server or computing device to be loaded or executed various programs.

[0307] In the context of the disclosed method, the pipeline receives input for post-processing, which may include a video data object and a target audio data object. The system is configured to generate new output video data objects that effectively replace certain regions (such as regions within the mouth area). The target audio data object may first be decomposed into timestamped audio tokens, which are mapped to phonemes and then to corresponding view positions. Effectively, each timestamped audio token can represent a mouth shape or mouth movement corresponding to the target audio data object.

[0308] Because the original video contains speech in the original language, it is necessary to automatically adjust the mouth movements and / or facial movements of individuals in the output video to match the target audio data object (e.g., the target language audio track). As described in this paper, performing this process manually is difficult and impractical, and this paper proposes a machine learning method that attempts to automatically generate replacement videos.

[0309] A first instance of the dedicated machine may include a server configured to generate alternative output video objects based on a set of untangled facial expressions and gestures when controlling the operation of a machine learning network. For example, the set of parameter instructions may be based on a specific viewpoint that temporally corresponds to a new mouth movement at a specific point in time, corresponding to a target mouth movement in the target language of the desired output audio for the output video object. Optionally, the set of parameter instructions may be extended with additional parameters representing residual parameters.

[0310] In this example, the machine learning network has two sub-networks: the first is a sound-to-lips machine learning model, and the second is a lip-to-image machine learning model. In this example, these two models interact to reconstruct frames, thereby creating a new output video data object. These two models can be used together in a coarse / fine reconstruction process, where an initial coarse frame can be refined to create a fine frame. During reconstruction, the model works with a mask frame, where repairs can be performed to replace specific portions of the image frame only according to the mask frame's regions (e.g., only on the mask portion).

[0311] In some embodiments, the output may be instructions for repair, which may be provided to downstream systems, or in other embodiments, the output may be a replacement area of ​​a masked portion or an entire replaced frame, depending on the system configuration. The pipeline computing component may receive the replaced output video or replacement frames, and in other embodiments, may evaluate the quality control of these frames or video portions thereof, for example, by indicating whether a frame or video portion is approved / unapproved. If a frame / video portion is unapproved, in another embodiment, the system may be configured to regenerate the specific portion, and the unapproval may be used for further training of the system. In some embodiments, an iterative process may be performed before providing the final output video data object to the next step in the post-processing pipeline, until there are no unapproved portions and all portions or frames have passed the quality control process.

[0312] The post-processing pipeline can have multiple processors or systems operating in parallel. For example, it can receive video in its original language, such as French. Audio tracks in Spanish, English, German, Korean, Chinese, Japanese, Malay, Indonesian, Swahili, etc., may also be desired. Each of these target audio tracks can be obtained, for example, through local voice talent, computer voice synthesis using translation programs, etc. The system can be tasked with in post-production to create multiple videos in parallel, where lip movements are modified to match each of these target audio tracks. Each generated video can then undergo a quality control process until a reviewer (e.g., a reviewer system or a human reviewer) is satisfied with the output.

[0313] This paper proposes variations of the computing architecture. For example, in an example embodiment, a single U-net is utilized, which has demonstrated strong performance in experimental analysis.

[0314] Variations of the masking method were also proposed, such as an improved mask that extends the masking region to the nose area instead of just below the nose, and the masking method was found to exhibit strong performance in experimental analysis.

[0315] The applicant notes that the described embodiments and examples are illustrative and not restrictive. Actual implementations of the features may combine some or all of the aspects, and the features described herein should not be considered as indications of future or existing product plans.

[0316] Although embodiments have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope. Furthermore, the scope of this application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods, and steps described herein.

[0317] As will be readily understood by those skilled in the art from this disclosure, existing or later-developed processes, machines, manufactures, material compositions, components, methods, or steps that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein can be utilized. Therefore, the appended embodiments are intended to encompass such processes, machines, manufactures, material compositions, components, methods, or steps within their scope.

[0318] It is understood that the examples described and shown above are intended to be illustrative only.

Claims

1. A computer-implemented system for generating an output video of lip-syncing, the system comprising: processor; Memory, coupled to the processor and storing processor-executable instructions, which, when executed, configure the processor to: Receive an initial video data object V, which includes multiple image frame data objects I; Extract the initial expression / viewpoint code set corresponding to the spectrogram segment in the initial audio data object A corresponding to the initial video data object V; Receive the set of view position parameters corresponding to the time synchronization of the target audio data object A'; and The initial video data object V and the time-synchronized view position parameter set are processed using a machine learning network to generate an output video data object V', wherein the initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized view position parameter set. The set of view position parameters corresponding to the time synchronization of the target audio data object A' is a set of parameters identified using multiple encoder networks, which are adapted to unwrap facial expressions based on multiple image characteristics. e and posture p ,as well as The plurality of encoder networks are trained iteratively to unwrap expression parameters from the plurality of image frame data objects I. e Posture parameters p and residual parameters r .

2. The system according to claim 1, wherein the initial video data object V is used to populate the viewpoint library of available viewpoints based on the initial expression / viewpoint code set.

3. The system of claim 2, wherein the processor is configured to compare the gaze positions identified in the target audio data object A' with the gaze position library of available gaze positions based on the initial expression / view position code set to identify a subset of gaze positions that are not in the gaze position library and need to be synthesized.

4. The system of claim 3, wherein the processor is configured to generate new expression / viewpoint codes corresponding to the subset of viewpoints not in the viewpoint library.

5. The system of claim 1, wherein the processor is configured to generate the output video data object V' by replacing at least one of the plurality of image frame data objects I with a replacement image frame data object I'.

6. The system of claim 1, wherein the processor is configured to generate the output video data object V' by replacing a portion of at least one of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to the mouth region.

7. The system of claim 1, wherein the processor is configured to iteratively generate the output video data object V' for each character present in the initial video data object V.

8. The system of claim 1, wherein the machine learning network comprises a generative adversarial network.

9. A computer-generated method for generating output video, the method comprising: Receive an initial video data object V, which includes multiple image frame data objects I; Extract the initial expression / viewpoint code set corresponding to the spectrogram segment in the initial audio data object A corresponding to the initial video data object V; Receive the set of view position parameters corresponding to the time synchronization of the target audio data object A'; and The initial video data object V and the time-synchronized view position parameter set are processed using a machine learning network to generate an output video data object V', wherein the initial mouth region in the initial video data object V has been replaced with a replacement mouth region generated based on the time-synchronized view position parameter set. The set of view position parameters corresponding to the time synchronization of the target audio data object A' is a set of parameters identified using multiple encoder networks, which are adapted to unwrap facial expressions based on multiple image characteristics. e and posture p ,as well as The plurality of encoder networks are trained iteratively to unwrap expression parameters from the plurality of image frame data objects I. e Posture parameters p and residual parameters r .

10. The method of claim 9, wherein the initial video data object V is used to populate a viewpoint library of available viewpoints based on the initial expression / viewpoint code set.

11. The method of claim 10, further comprising comparing the gaze positions identified in the target audio data object A' with a library of available gaze positions based on the initial set of expression / view position codes to identify a subset of gaze positions not in the library, and generating new expression / view position codes corresponding to the subset of gaze positions not in the library.

12. The method of claim 9, further comprising generating the output video data object V' by entirely replacing at least one of the plurality of image frame data objects I with a replacement image frame data object I'.

13. The method of claim 9, further comprising generating the output video data object V' by replacing a portion of at least one of the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I', wherein the portion corresponds to the mouth region.

14. The method of claim 9, further comprising iteratively generating the output video data object V' for each character present in the initial video data object V.

15. A non-transitory computer-readable medium storing a set of machine-interpretable instructions that, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 9 to 14.

Images