Video figure mouth shape synchronization method based on deep learning

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and particularly relates to a video figure mouth shape synchronization method based on deep learning. According to the method, high-precision and low-delay lip action generation is achieved through multi-modal feature fusion, a generative adversarial network (GAN) and a differentiable rendering technology, the method is suitable for scenes such as film and television post-production, virtual reality (VR) real-time interaction, voice-driven animation generation and multi-language video translation, the synchronization error of the method on a standard data set is reduced by 62.5%, and the real-time performance of the method is improved. The method supports 30fps real-time processing, has strong noise robustness and multilingual adaptability, and can be widely applied to film and television production, virtual reality and real-time interaction scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically to a deep learning-based method for lip-syncing video characters. This method achieves high-precision, low-latency lip movement generation through multimodal feature fusion, generative adversarial networks (GANs), and differentiable rendering technology. It is suitable for scenarios such as film and television post-production, real-time virtual reality (VR) interaction, voice-driven animation generation, and multilingual video translation. Background Art

[0002] In the field of video production, especially in animation, film and television post-dubbing, and virtual character interaction, how to achieve accurate synchronization of lip movements and voice content has always been a technical challenge; traditional lip-sync animation production requires a lot of manpower and time, and the results are often unsatisfactory. However, with the continuous development of artificial intelligence technology, this problem is being gradually overcome. AI automatically generates lip movements and achieves automatic synchronization of video lip movements, which is becoming a revolutionary technology in the field of video production. AI's automatic lip-syncing technology mainly relies on technologies such as deep learning and natural language processing.

[0003] Deep learning-based video lip synchronization technology mainly relies on the following two types of methods:

[0004] 1. Traditional method based on key point matching:

[0005] This method detects the motion trajectory of key lip points (such as lip peaks and lip corners) in the video and matches them with the spectral features of the input audio through dynamic time warping (DTW) or linear interpolation. This method has significant drawbacks:

[0006] Sensitive to lighting and posture: lighting changes or head deflection can cause keypoint detection errors;

[0007] Insufficient modeling of nonlinear relationships: Manually designed features are difficult to capture the subtle differences in complex articulatory movements (such as bilabial and alveolar sounds);

[0008] Poor real-time performance: The key point matching algorithm has high computational complexity and cannot meet the needs of real-time interaction.

[0009] 2. Statistical model method based on template synthesis:

[0010] This method uses a library of pre-recorded lip movement templates and combines them with a hidden Markov model (HMM) or a Gaussian mixture model (GMM) to select the optimal template for synthesis. However, its limitations include:

[0011] The template library has limited coverage: it is difficult to cover multiple languages, dialects, and special pronunciation scenarios;

[0012] The synthesis effect is stiff: template splicing results in unnatural movement transitions;

[0013] Weak generalization ability: poor adaptability to noisy environments or low-quality audio.

[0014] In addition, the existing technologies also have the following common problems:

[0015] It is difficult to strike a balance between accuracy and real-time performance: Traditional methods sacrifice processing speed when improving accuracy, and vice versa;

[0016] Insufficient multimodal feature fusion: The correlation between audio and visual features is not fully explored;

[0017] Lack of end-to-end optimization: Most solutions rely on staged processing, which leads to error accumulation. Summary of the Invention

[0018] The purpose of the present invention is to address the shortcomings of the prior art mentioned in the background art, and to propose a video character lip synchronization method based on deep learning, aiming to achieve the following goals:

[0019] High-precision synchronization: Deep neural networks are used to model the nonlinear mapping between lip movements and audio;

[0020] Low-latency processing: Optimizes model architecture and inference process to support real-time video stream generation;

[0021] Strong generalization ability: adaptable to multiple languages, noisy environments and complex head posture changes.

[0022] In order to achieve the above object, the present invention adopts the following technical solutions:

[0023] A method for lip syncing of video characters based on deep learning, comprising the following steps:

[0024] Extract acoustic features of input audio, including MFCC coefficients, phoneme sequence and fundamental frequency;

[0025] Extract the spatiotemporal dynamic features of the lips in the video frames through 3D convolutional neural network;

[0026] Adopt multi-head self-attention mechanism to align and fuse acoustic and visual features;

[0027] Generate lip keypoint sequences synchronized with audio based on generative adversarial networks (GANs);

[0028] Use a differentiable rendering module to map key points to natural lip images;

[0029] The temporal coherence of the generated video is optimized through the optical flow loss function.

[0030] As a further feature of the present invention, the extraction of the acoustic features adopts MFCC analysis with a frame length of 25ms and a step length of 10ms, and splices phoneme one-hot coding.

[0031] As a further step in the present invention, the weight distribution formula of the multi-head self-attention mechanism is: Among them, Q is the audio feature matrix, K is the visual feature matrix, and d=512.

[0032] As a further step in the present invention, the generator of the generative adversarial network (GAN) is a U-Net structure, which includes 5 layers of downsampling and 5 layers of upsampling, and jump connections transmit low-level features; the discriminator is a PatchGAN architecture, which outputs the authenticity probability of local image blocks.

[0033] As a further step in the present invention, the differentiable rendering module calculates the coordinates of key points through bilinear interpolation and generates a lip image by combining local texture synthesis. The texture library contains 100 lip texture templates.

[0034] As a further step in the present invention, the optical flow loss function calculates the difference in optical flow fields of adjacent frames, and the formula is: Among them, φ is the optical flow estimation network, I t For real frames, To generate a frame.

[0035] As a further feature of the present invention, the method supports real-time processing, single-frame delay ≤ 20ms, and the hardware platform is an embedded GPU device.

[0036] As a further aspect of the present invention, a video lip-sync system includes an audio input module configured to receive and pre-process an audio signal;

[0037] a video input module configured to capture and crop a lip region;

[0038] A multimodal fusion module, which implements the feature alignment and fusion described in claims 1-7;

[0039] Lip generation module, which synthesizes key point sequences based on GAN;

[0040] Rendering output module generates and optimizes the final video stream.

[0041] As a further aspect of the present invention, a video lip synchronization system is deployed on an edge computing device to support real-time processing of 4K resolution video.

[0042] As a further aspect of the present invention, a computer-readable storage medium stores program instructions, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 7.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] Improved accuracy: On a standard dataset (LRS3), the synchronization error (DTW distance) between lip movements and audio is 0.12, a 62.5% reduction compared to the traditional method (0.32);

[0045] Real-time performance: single frame processing time ≤ 20ms, supports 30fps real-time generation;

[0046] Generalization ability: On a noisy (SNR = 10dB) and multi-language (Chinese, English, Spanish) test set, the synchronization error fluctuation is ≤10%. DETAILED DESCRIPTION

[0047] The present invention may be more readily understood by referring to the following detailed description of preferred embodiments of the present invention and the included Examples. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention pertains. In the event of a conflict, the definitions in this specification shall prevail.

[0048] The singular includes plural references unless the context clearly dictates otherwise. "Optional" or "either" means that the subsequently described event or incident can or cannot occur, and that the description includes instances where the event occurs and instances where it does not.

[0049] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0050] This paper discloses a deep learning-based method for lip-syncing video characters. By integrating multimodal feature fusion, generative adversarial networks, and differentiable rendering techniques, it achieves high-precision, low-latency lip movement generation. This method reduces synchronization error by 62.5% on a standard dataset, supports real-time processing at 30fps, and exhibits strong noise robustness and multilingual adaptability. It is widely applicable to film and television production, virtual reality, and real-time interactive scenarios.

[0051] The core steps are as follows:

[0052] 1. Multimodal feature extraction:

[0053] Audio features: Extract MFCC (Mel-frequency cepstral coefficients), phoneme sequence and fundamental frequency (F0) of the input audio;

[0054] Visual features: The spatiotemporal dynamic features of the lips in video frames are extracted through a 3D convolutional neural network (3D-CNN).

[0055] 2. Cross-modal feature fusion:

[0056] The multi-head self-attention mechanism is used to align the temporal relationship between audio and visual features. The weight calculation formula is: Among them, Q is the audio feature matrix, K is the visual feature matrix, and d is the feature dimension;

[0057] The fused features are then used for time series modeling through bidirectional LSTM to capture long-term dependencies.

[0058] 3. Lip movement generation:

[0059] Generative Adversarial Network (GAN): The generator uses a U-Net structure, inputs fused features and outputs a sequence of lip keypoints; the discriminator uses a PatchGAN architecture to distinguish generated and real keypoints;

[0060] Differentiable rendering module: Based on bilinear interpolation and local texture synthesis, key points are mapped to natural lip images.

[0061] 4. Post-processing optimization:

[0062] Temporal consistency constraint: Introducing the optical flow loss function to ensure the temporal coherence of the generated video frames;

[0063] Super-resolution enhancement: Improve the resolution and detail clarity of the output video through a lightweight ESRGAN network.

[0064] A method for lip syncing of video characters based on deep learning, comprising the following steps:

[0065] Extract acoustic features of input audio, including MFCC coefficients, phoneme sequence and fundamental frequency;

[0066] Extract the spatiotemporal dynamic features of the lips in the video frames through 3D convolutional neural network;

[0067] Adopt multi-head self-attention mechanism to align and fuse acoustic and visual features;

[0068] Generate lip keypoint sequences synchronized with audio based on generative adversarial networks (GANs);

[0069] Use a differentiable rendering module to map key points to natural lip images;

[0070] The temporal coherence of the generated video is optimized through the optical flow loss function.

[0071] Acoustic features are extracted using MFCC analysis with a frame length of 25ms and a step length of 10ms, and the phoneme one-hot encoding is concatenated.

[0072] The weight distribution formula of the multi-head self-attention mechanism is Among them, Q is the audio feature matrix, K is the visual feature matrix, and d=512.

[0073] The generator of the generative adversarial network (GAN) is a U-Net structure, which consists of 5 layers of downsampling and 5 layers of upsampling, and skip connections transmit low-level features; the discriminator is a PatchGAN architecture, which outputs the authenticity probability of local image blocks.

[0074] The differentiable rendering module calculates the coordinates of key points through bilinear interpolation and combines local texture synthesis to generate lip images. The texture library contains 100 lip texture templates.

[0075] The optical flow loss function calculates the difference in optical flow fields between adjacent frames, and the formula is: Among them, φ is the optical flow estimation network, I t For real frames, To generate a frame.

[0076] The method supports real-time processing with a single-frame delay of ≤20ms, and the hardware platform is an embedded GPU device.

[0077] A video lip-sync system, an audio input module configured to receive and pre-process an audio signal;

[0078] a video input module configured to capture and crop a lip region;

[0079] Multimodal fusion module, which implements the feature alignment and fusion in claims 1-7;

[0080] Lip generation module, which synthesizes key point sequences based on GAN;

[0081] Rendering output module generates and optimizes the final video stream.

[0082] A video lip-sync system deployed on edge computing devices supports real-time processing of 4K resolution videos.

[0083] A computer-readable storage medium stores program instructions, wherein the program instructions implement the method according to any one of claims 1 to 7 when executed by a processor.

[0084] Example 1: Lip synchronization of a standard video dataset

[0085] 1. Data preparation:

[0086] Dataset: The LRS3 public dataset is used, which contains 100 hours of annotated audio-video pairs;

[0087] Preprocessing:

[0088] Audio: 16kHz sampling rate, extracting MFCC features (40 dimensions) with a frame length of 25ms and a step size of 10ms;

[0089] Video: The lip area is cropped and scaled to 256×256 resolution, with a frame rate of 25fps.

[0090] 2. Model training:

[0091] Network architecture:

[0092] 3D-CNN: 5 convolutional layers, channels [64, 128, 256, 512, 512], kernel size 3 × 3 × 3;

[0093] Number of attention heads: 8, feature dimension d = 512;

[0094] LSTM: hidden layer dimension 1024, bidirectional structure;

[0095] Generator: U-Net, skip connection to reduce information loss;

[0096] Training parameters:

[0097] Optimizer: Adam (learning rate 1e-4, β1=0.5, β2=0.999);

[0098] Loss function: L1 reconstruction loss (weight 0.7) + adversarial loss (weight 0.2) + optical flow loss (weight 0.1);

[0099] Batch size: 16, training epochs: 100.

[0100] 3. Test results:

[0101] Synchronization error (DTW distance): 0.12 (traditional method is 0.32);

[0102] Generated video quality: PSNR = 32.5dB, SSIM = 0.91;

[0103] Subjective evaluation: 50 subjects gave an average rating of 4.6 / 5.0 (naturalness).

[0104] Example 2: Lip synchronization of real-time video streams

[0105] 1. System deployment:

[0106] Hardware platform: NVIDIA Jetson AGX Xavier;

[0107] Software optimization: Convert the PyTorch model to the TensorRT engine and enable FP16 quantization;

[0108] Input sources: real-time audio stream (microphone input, 16kHz) and camera video stream (30fps).

[0109] 2. Performance testing:

[0110] End-to-end latency: 18ms (including feature extraction, fusion, generation, and rendering);

[0111] Resource usage: GPU utilization ≤ 75%, memory usage 2.5GB;

[0112] Noise environment test: Adding Gaussian white noise (SNR=10dB), the synchronization error only increases by 8%.

[0113] Example 3: Multilingual Lip Synchronization Adaptation

[0114] 1. Data expansion:

[0115] New Chinese (AISHELL-3) and Spanish (MultilingualLibriSpeech) datasets added;

[0116] Phonemic Transcription: Align audio and text using the MontrealForced Aligner tool.

[0117] 2. Cross-language adaptation:

[0118] Fine-tuning strategy: Freeze the 3D-CNN and rendering modules, and only train the attention layer and LSTM;

[0119] Results: On an untrained language (French), the synchronization error is 0.15, showing strong generalization ability.

[0120] The examples referred to herein are merely illustrative and serve to illustrate some features of the method of the present invention, and the appended claims are intended to claim the widest possible scope that can be imagined, and the embodiments presented herein are merely illustrative of selected implementation methods according to the combination of all possible embodiments. Therefore, it is the applicant's intention that the appended claims are not limited by the selection of examples illustrating the features of the present invention. Some numerical ranges used in the claims also include subranges therein, and changes in these ranges should also be interpreted as being covered by the appended claims where possible.

Claims

1. A video character lip synchronization method based on deep learning, characterized in that: The following steps are included: Extract acoustic features of input audio, including MFCC coefficients, phoneme sequence and fundamental frequency; Extract the spatiotemporal dynamic features of the lips in the video frames through 3D convolutional neural network; Adopt multi-head self-attention mechanism to align and fuse acoustic and visual features; Generate lip keypoint sequences synchronized with audio based on generative adversarial networks (GANs); Use a differentiable rendering module to map key points to natural lip images; The temporal coherence of the generated video is optimized through the optical flow loss function.

2. A method for lip syncing of video characters based on deep learning according to claim 1, characterized in that: The acoustic features are extracted by MFCC analysis with a frame length of 25ms and a step length of 10ms, and the phoneme one-hot coding is performed.

3. The method for lip synchronization of video characters based on deep learning according to claim 1, characterized in that: The weight distribution formula of the multi-head self-attention mechanism is: Among them, Q is the audio feature matrix, K is the visual feature matrix, and d=512.

4. The method for lip synchronization of video characters based on deep learning according to claim 1, wherein: The generator of the generative adversarial network (GAN) is a U-Net structure, which includes 5 layers of downsampling and 5 layers of upsampling, and skip connections to transfer low-level features; The discriminator is a PatchGAN architecture that outputs the authenticity probability of a local image block.

5. The method for lip synchronization of video characters based on deep learning according to claim 1, wherein: The differentiable rendering module calculates the coordinates of key points through bilinear interpolation and generates lip images by combining local texture synthesis. The texture library contains 100 lip texture templates.

6. The method for lip synchronization of video characters based on deep learning according to claim 1, characterized in that: The optical flow loss function calculates the difference in optical flow fields between adjacent frames, and the formula is: Among them, φ is the optical flow estimation network, I t For real frames, To generate a frame.

7. The method for lip synchronization of video characters based on deep learning according to claim 6, characterized in that: The method supports real-time processing with a single-frame delay of ≤20ms, and the hardware platform is an embedded GPU device.

8. A video lip-sync system according to any one of claims 1 to 7, characterized in that: an audio input module configured to receive and pre-process an audio signal; a video input module configured to capture and crop a lip region; A multimodal fusion module, which implements the feature alignment and fusion described in claims 1-7; Lip generation module, which synthesizes key point sequences based on GAN; Rendering output module generates and optimizes the final video stream.

9. A video lip-sync system according to claim 8, characterized in that: Deployed on edge computing devices, it supports real-time processing of 4K resolution videos.

10. The computer-readable storage medium according to any one of claims 8 to 9, wherein: Program instructions are stored, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images