Three-dimensional speaker facial animation generation method based on text driving

Abstract

The invention discloses a three-dimensional speaker facial animation generation method based on text driving. A face animation generation network is constructed and specifically comprises an audio encoder, a style encoder, an audio-to-action adapter, an action decoder and an audio decoder, texts are used as input of the audio encoder, the audio encoder is connected with the audio-to-action adapter, the audio encoder is connected with the audio decoder, and the action decoder is connected with the audio decoder. And a result obtained after the output of the audio-to-action adapter and the output of the style encoder are fused is recorded as a style action code, the action decoder is used for decoding into a 3DMM parameter set according to the style action code, and the audio decoder is used for synthesizing voice synchronous with the face animation according to the continuous voice latent representation output by the audio encoder. According to the method, three-dimensional speaker animations with controllable styles and voice output are directly generated from plain text input, high synchronization precision and cross-scene adaptive capacity are achieved, and the efficient and convenient facial animation generation method is provided.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer graphics and multi-modal generation, and particularly relates to a three-dimensional speaker face animation generation method based on text driving. BACKGROUND

[0002] Realistic face animation generation technology has attracted widespread attention in the fields of video games, virtual reality and metaverse. High-quality face animation generation generally includes: first, collecting visual input data (such as RGB, RGB-D or infrared images) of a person; then converting the face performance into a parameterized representation, such as vertex displacement or a three-dimensional morphable model (3DMM); finally, using a skeletal, blendshape or hybrid-driven face binding system to achieve accurate deformation and expression reproduction.

[0003] Existing speech-driven face animation generation methods have made progress in naturalness and synchronization accuracy, but they generally rely on large-scale paired audio-visual training data, which is costly to collect and dependent on specific speakers. In contrast, text-driven speaker animation generation has the advantages of easy access and easy standardization, but related research is less, especially in the field of three-dimensional parameter-based animation generation, which still lacks fine control of identity, emotion and speaking style.

[0004] Early methods mostly adopt non-parametric strategies, directly driving mouth shape generation through speech signals (such as spectral envelope, prosodic features), mapping phonemes or phoneme combinations to corresponding pronunciation poses (visemes). However, this type of method has limitations in generating natural and expressive mouth shapes. In recent years, end-to-end deep learning methods have improved feature extraction capabilities with pre-trained audio encoders (such as Wav2Vec2), and a variety of three-dimensional face animation generation frameworks have emerged, such as MeshTalk, FaceFormer, SelfTalk and FaceTalk based on diffusion models. These methods have improved in expressiveness and temporal consistency, and unified models that can integrate multiple datasets (such as UniTalker) have also appeared. However, these methods are all audio-driven and have some drawbacks in terms of ease of use.

[0005] To achieve emotion and style controllable three-dimensional animation generation, some research has introduced emotion modeling modules, such as EmoTalk and EmoVOCA, which use a dual-encoder structure to disentangle emotion and content and achieve label-driven emotion and intensity control. However, most current text-driven methods are only applied to two-dimensional portrait generation and lack direct control over dynamic three-dimensional parameters, limiting their application in virtual character animation and real-time interaction scenarios.

[0006] In summary, the prior art still has deficiencies in directly driving controllable three-dimensional speaker animation generation by text, and urgently needs a unified framework that can simultaneously synthesize speech and three-dimensional facial animation under the condition of only relying on text input, and realize accurate control of identity, emotion and speaking style. SUMMARY

[0007] The present application aims to solve the problems of existing three-dimensional speaker facial animation generation, such as relying on large-scale paired audio-video data, strong dependence on speakers, insufficient style and emotion control ability, and difficulty in directly outputting three-dimensional parameters that can be used for downstream driving, and proposes a unified framework that can simultaneously synthesize speech and three-dimensional facial animation only relying on text input, and provides controllable generation ability for identity, emotion and speaking style.

[0008] The technical solutions of the present application are as follows:

[0009] I. A three-dimensional speaker facial animation generation method based on text driving

[0010] Step 1: Construct a face animation generation network;

[0011] Step 2: After training the face animation generation network using the first training set, obtain the trained face animation generation network;

[0012] Step 3: Obtain the target text and input it into the trained face animation generation network, and the network outputs synchronous face animation and speech waveform.

[0013] The face animation generation network includes an audio encoder, a style encoder, an audio-to-motion adapter, a motion decoder, and an audio decoder. The text is input into the audio encoder together. The audio encoder is connected to the audio-to-motion adapter. The audio encoder is connected to the audio decoder. The output of the audio-to-motion adapter is fused with the output of the style encoder, and the result is recorded as style motion encoding and used as the input of the motion decoder. The motion decoder is used to decode the 3DMM parameter set according to the style motion encoding, thereby outputting the face animation. The audio decoder is used to synthesize the speech synchronized with the face animation according to the continuous speech latent representation output by the audio encoder.

[0014] In the training process of the face animation generation network, the audio decoder and the style encoder are frozen.

[0015] In step 2, before training the audio encoder, audio-to-motion adapter, and motion decoder of the face animation generation network, the second training set is used to pre-train the style encoder to obtain the pre-trained style encoder.

[0016] In step 3, the input of the trained face animation generation network also includes one or both of the reference audio and the target style label.

[0017] The audio-to-action adapter is composed of a Transformer context network layer in Wav2Vec2, which is used to perform mapping between a speech frame rate and an action frame rate, output an action feature sequence, and is denoted as an initial action encoding.

[0018] The face animation generation network further includes a post-processing unit configured to perform intensity scaling or interpolation and smoothing processing on the output of the style encoder to obtain a post-processed style embedding, and fuse the post-processed style embedding with the output of the audio-to-action adapter.

[0019] II. A computer device

[0020] The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0021] III. A computer readable storage medium

[0022] The medium stores a computer program, and the computer program implements the steps of the method when executed by a processor.

[0023] IV. A computer program product

[0024] The product includes a computer program / instruction, which implements the steps of the method when executed by a processor.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] Text-driven expression animation generation: the present application directly generates a three-dimensional animation parameter sequence that can drive a standard model from text, reduces dependence on paired audio and video data, and improves the reusability of the generation link;

[0027] Controllable style expression: the present application uniformly models identity and emotion through a style encoder based on contrast learning, and realizes style intensity adjustment, interpolation and migration;

[0028] Cross-modal alignment: the audio-to-action adapter proposed in the present application establishes a robust mapping between speech and action, improving the consistency of lip shapes and speech;

[0029] Heterogeneous data fusion: the present application is based on multi-dataset joint training under mask supervision, which can utilize multiple annotation formats at the same time and improve the generalization ability;

[0030] Engineering interface friendly: the present application outputs industry common parameters such as ARKit / FLAME, which is convenient for landing real-time driving and offline rendering. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 This is a flowchart illustrating the process architecture of a text-driven method for generating 3D speaker facial animations.

[0032] Figure 2 This is a schematic diagram of the training of the style encoder.

[0033] Figure 3 This is a schematic diagram of the audio-to-motion adapter. Detailed Implementation

[0034] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] This invention proposes a text-driven method for generating 3D speaker facial animation, which enables the direct generation of 3D speaker animation and speech output with controllable style from plain text input. For example... Figure 1 As shown, the method includes the following steps:

[0037] Step 1: Construct a face animation generation network; the overall structure of the face animation generation network adopts a pipeline framework of encoding-adaptation-decoding.

[0038] The face animation generation network comprises an audio encoder, a style encoder, an audio-to-motion adapter, a motion decoder, and an audio decoder. Text and reference audio are used as input to the audio encoder, with the reference audio having the same sampling rate as the original audio and a duration not exceeding a certain number of seconds. The audio encoder outputs a continuous latent speech representation containing the speech style of the reference audio. If no reference audio is available, the audio encoder outputs a continuous latent speech representation including random speech styles. Target style labels are used as input to the style encoder to characterize identity and emotional attributes. If no target style label is available, the style encoder outputs a default style embedding encoding. The audio encoder is connected to the audio-to-motion adapter, and the audio encoder is also connected to the audio decoder. The output of the audio-to-motion adapter and the output of the style encoder are fused together, resulting in a style-motion code, which serves as the input to the action decoder. The action decoder decodes the style-motion code into a 3DMM parameter set (i.e., a 3D deformable face model parameter set, such as an ARKit coefficient sequence or a FLAME parameter sequence), thereby outputting a facial animation. The audio decoder synthesizes speech synchronized with the facial animation based on the continuous speech latent representation output by the audio encoder, thus obtaining a synchronized 3D facial parameter sequence and speech waveform. The final 3D parameter sequence and optional speech waveform are encapsulated to generate a data packet containing information such as timestamps, parameter values, and format mappings. The output format can be ARKit, FLAME, or standard 3D formats such as FBX or GLTF. It can directly drive the rendering engine as a plugin or be saved as an offline file for batch rendering.

[0039] The face animation generation network also includes a post-processing unit for intensity scaling, interpolation, and smoothing of the style encoder output. This post-processed style embedding is then fused with the audio-to-motion adapter output. Style intensity adjustment can directly affect the style vector or parameter output, while interpolation enables smooth transitions between different identities or emotions. Prosodic consistency optimization utilizes cues such as pauses and stresses to smooth out transitions at sentence / comma boundaries, reducing abrupt changes. Temporal smoothing employs exponential moving average filtering to reduce lip-sync jitter.

[0040] The text input to the network is first processed by regularization, including expansion of numbers, units, and abbreviations, as well as multilingual detection and routing. After word segmentation or BPE (Byte Pair Encoding), a standardized text token sequence is obtained.

[0041] Optionally, the audio encoder is a TTS encoder. The audio decoder is a TTS decoder.

[0042] For TTS encoders, a decoding-based pre-trained multilingual TTS model based on GPT-2 is preferred. The input text is autoregressively encoded to output a continuous speech latent representation (such as Mel-spectral features compatible with HiFi-GAN), and speaker-related conditional information can be extracted by combining it with reference audio.

[0043] Style Encoder: Obtains identity-emotion joint style embeddings through contrastive learning to control the style representation of facial animations; on datasets lacking emotion annotations, style labels can revert to identity labels.

[0044] Optionally, such as Figure 2 As shown, the style encoder employs a contrastive learning-based mechanism to extract individualized style features from speech or multimodal inputs, thereby improving the performance of speech synthesis or 3D face-driven models. The style encoder receives speech signals or other relevant modal data and transforms them into standardized feature representations, such as Mel-spectrograms, implicit speech embedding vectors, or other frequency / time domain features. The style encoder performs multi-level modeling of the input signal through an encoder-only Transformer module to obtain a high-dimensional representation vector related to the speaker's style, i.e., a low-dimensional style embedding vector. This embedding vector is used to characterize non-linguistic attributes of speech, such as prosody, timbre, and emotion.

[0045] An audio-to-motion adapter is used to map latent speech representations to facial motion features.

[0046] Optionally, such as Figure 3 As shown, the audio-to-action adapter consists of a Transformer context network layer in Wav2Vec2. This Transformer context network layer performs mapping between the speech frame rate and the action frame rate, outputting an action feature sequence as the initial action code, thus achieving feature alignment and temporal modeling between speech and action modalities. Action feature sequence M 1:V Satisfy M 1:V =g A2M (Resample(A 1:U ), where A 1:U For an acoustic feature sequence (i.e., a continuous speech latent representation), each frame contains Mel spectral features as intermediate acoustic representations. `Resample()` is a linear transformation function that scales the acoustic feature sequence to the desired action feature sequence length. A2M( ) represents the audio-to-action adapter module. This invention uses a cropped version of the wav2vec 2 model as the bridging unit, retaining only the Transformer context network layer and removing the feature encoder and quantization module to reduce latency and computation.

[0047] This adapter can smoothly map audio codes in Mel spectrum form to motion codes required for action generation, enabling cross-modal adaptation in speech-driven action generation.

[0048] Table 1 Ablation experiments of the audio-to-motion adapter module

[0049]

[0050] Ablation analysis was performed on the audio-to-motion adapter, and the ablation experimental results are shown in Table 1. Here, EVE represents the facial expression vertex error (lower is better), and LVE represents the lip vertex error (lower is better). Table 1 shows that removing the audio-to-motion adapter leads to a significant increase in LVE error, while the EVE error remains unchanged. This indicates that the module captures the mapping relationship related to lip movements in the generated facial animation and can effectively drive fine-grained lip movements. Therefore, the adapter module plays an irreplaceable role in achieving accurate speech-driven pronunciation and maintaining the temporal synchronization between audio and facial movements.

[0051] Unlike traditional discrete-output TTS systems, this invention preserves a continuous acoustic feature sequence to ensure the smoothness and temporal consistency of subsequent action generation.

[0052] Motion decoder: Decodes motion features into 3D deformable model (3DMM) parameters, including but not limited to ARKit expression coefficients and FLAME shape / expression / pose parameters, which facilitates direct driving of face binding and mesh deformation.

[0053] The action decoder backbone preferably uses a temporal convolutional network, and cross-frame self-attention can be optionally added to capture long-term dependencies. The action decoder adopts a multi-output head structure, with the temporal convolutional network connected to multiple MLP networks. Each MLP serves as an output head, and different output heads correspond to different 3D parameter types, such as ARKit blendshape coefficients and FLAME parameters.

[0054] Speech generation decoder: Employs a frozen HiFi-GAN vocoder to restore the acoustic features output by the TTS encoder into a speech waveform (can be frozen during training and used only for inference).

[0055] Step 2: After training the face animation generation network using the first training set, the trained face animation generation network is obtained;

[0056] This invention employs a two-stage training process:

[0057] Phase 1:

[0058] A pre-trained style encoder is obtained by pre-training using a second training set. Specifically, the second training set includes Unitalker-Speaker and Unitalker-Song, containing speech-emotion label pairs of speaker and emotional states. The input to the style encoder is a speech segment with speaker identity and emotional labels. InfoNCE contrastive learning loss is used to supervise the style encoder in generating specific style codes for audio of different styles, ensuring that samples of the same identity-emotion style are clustered in the embedding space and that different styles are separated. When the dataset lacks emotional labels, positive and negative pairs are constructed only by identity, aiming to generate robust and informative style priors to guide subsequent action generation. Style intensity can be adjusted or transitions between different styles can be made through amplitude scaling or vector interpolation, and zero-shot style transfer is supported. Positive samples come from different speech segments under the same speech style or the same speaker, with the goal of maintaining high similarity in the embedding space; negative samples come from speech segments under different styles or different speakers, with the goal of maintaining low similarity in the embedding space. By maximizing the similarity between positive samples and minimizing the similarity between negative samples, the style features are distinguished and compressed.

[0059] Table 2 Ablation Experiment Results of Style Encoders

[0060]

[0061] Ablation analysis was performed on the style encoder, and the ablation experiment results are shown in Table 2. Table 2 shows that when the style encoder is removed, LVE (Lip Vertex Error, lower is better) and EVE (Expression Vertex Error, lower is better) increase significantly. This indicates that the style encoder plays a crucial role in capturing fine-grained speech style and emotional changes.

[0062] The second phase (multi-task joint training): In this phase, the TTS encoder, audio-to-motion adapter, and motion decoder are jointly trained, while the HiFi-GAN vocoder remains frozen. The training objective covers both speech synthesis and motion generation. The first training set includes multiple datasets such as RAVDESS, Faceforensics++, Unitalker-Speech, and Unitalker-Song for joint training to improve model performance and generalization ability.

[0063] Speech synthesis task: Apply TTS loss to the TTS encoder, and apply mean squared error loss to the continuous speech latent representation obtained by the TTS encoder and the corresponding ground truth labels to obtain the optimization function L. TTS Minimize the prediction error for acoustic features (such as Mel spectra).

[0064] Motion generation task: Utilizing multiple labeled animation datasets, including ARKit blendshape parameters and FLAME model parameters. b and L θ These correspond to ARKit's blendshape loss and FLAME's vertex loss, respectively, and a scale scaling factor is added to normalize the scale.

[0065] The total loss function for the second stage is L = L θ +αL b +βL TTS Here, α and β are two weighting coefficients used to balance the importance of speech synthesis and action generation tasks. In this embodiment, α is 0.0001 and β is 0.1. During joint training on multiple datasets, a mask supervision strategy is adopted: if a sample lacks a certain type of annotation (such as no FLAME parameter or no ARKit parameter), the corresponding item is not included in the loss of that sample, thereby fusing heterogeneous data.

[0066] Step 3: Obtain the target text and input it into the trained face animation generation network. The network outputs synchronized face animation and speech waveforms.

[0067] In one feasible implementation, the input to the trained face animation generation network may also include one or both of reference audio and target style labels.

[0068] This invention directly outputs 3D parameters (ARKit, FLAME, etc.) that can be used for driving, and can be seamlessly integrated into existing engines or DCC tools (such as real-time driving based on blendshape, and offline rendering based on mesh). It can also output synchronized voice for lip-syncing and content playback.

[0069] This invention also proposes a text-driven 3D speaker facial animation generation system, the system comprising:

[0070] The text acquisition unit is used to acquire the input target text;

[0071] The style information acquisition unit is used to acquire one or both of the reference audio and the target style tag.

[0072] The animation generation unit stores the face animation generation network and its corresponding training results. It generates a corresponding synchronized 3D facial parameter sequence and speech waveform based on the acquired target text, reference audio, and target style labels, and then displays and plays them on the screen.

[0073] To verify the superiority of the face animation generation network (Text2Talker) of this invention, quantitative analysis was performed on multiple datasets, and comparisons were made with state-of-the-art methods. In the baseline mode, Text2Talker, consistent with the training phase, additionally inputs a target audio length cue to ensure consistency of evaluation results.

[0074] The present invention uses the following quantitative indicators (in millimeters):

[0075] EVE (Eye Vertex Error): The maximum L2 distance between the vertex of the eye and forehead regions, used to characterize facial movements related to emotions;

[0076] LVE (Lip Vertex Error): The maximum L2 distance between vertices of the oral cavity region, used to reflect the accuracy of speech-related lip movements.

[0077] Table 3 Comparison results of the present invention on the 3D-ETF (RAVDESS) dataset

[0078]

[0079] As shown in Table 3, compared with methods such as VOCA, FaceFormer, MeshTalk, EmoTalk, and UniTalker, this invention achieves the best performance in both EVE and LVE (EVE = 1.214 × 10⁻⁻⁴). 5 LVE = 1.306 × 10⁻ 5 This invention significantly outperforms all previous methods. Experimental results demonstrate that the present invention can generate more expressive facial animations and achieve higher accuracy in speech-driven lip-sync. This performance improvement is primarily attributed to the proposed style encoder and audio-to-motion adapter, the effectiveness of which was further validated in ablation experiments.

[0080] Through the above technical solution, the present invention can generate a 3D facial animation that is synchronized with speech and has a controllable style and emotion when only text is input, and the output is a universally applicable and driveable parameter, which can be directly integrated into existing animation production and real-time driving systems.

[0081] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A text-driven based three-dimensional speaker face animation generation method, characterized in that, The method comprises the following steps: Step 1: constructing a face animation generation network; Step 2: obtaining a trained face animation generation network after training the face animation generation network using a first training set; Step 3: obtaining a target text and inputting the target text into the trained face animation generation network, and outputting a synchronous face animation and a voice waveform from the network.

2. The text-driven 3D speaker face animation generation method according to claim 1, wherein, The face animation generation network comprises an audio encoder, a style encoder, an audio-to-motion adapter, a motion decoder, and an audio decoder, wherein the text is input into the audio encoder together with audio, the audio encoder is connected to the audio-to-motion adapter, the audio encoder is connected to the audio decoder, the output of the audio-to-motion adapter is fused with the output of the style encoder, and the result is recorded as style motion encoding and used as the input of the motion decoder, the motion decoder is used to decode the 3DMM parameter set according to the style motion encoding, thereby outputting a face animation, and the audio decoder is used to synthesize a voice synchronized with the face animation according to the continuous voice latent representation output by the audio encoder.

3. The text-driven 3D speaker face animation generation method according to claim 1, wherein, During the training process of the face animation generation network, the audio decoder and the style encoder are frozen.

4. The text-driven 3D speaker face animation generation method according to claim 3, wherein, In step 2, before training the audio encoder, the audio-to-motion adapter, and the motion decoder of the face animation generation network, the style encoder is pre-trained using a second training set to obtain a pre-trained style encoder.

5. The text-driven 3D speaker face animation generation method according to claim 1, wherein, In step 3, the input of the trained face animation generation network further comprises one or both of a reference audio and a target style label.

6. The text-driven 3D speaker face animation generation method according to claim 2, wherein, The audio-to-motion adapter is a Transformer context network layer in Wav2Vec2, which is used to perform mapping between the voice frame rate and the motion frame rate, output a motion feature sequence, and record the motion feature sequence as initial motion encoding.

7. The text-driven 3D speaker face animation generation method according to claim 2, wherein, The face animation generation network further comprises a post-processing unit for performing intensity scaling or interpolation and smoothing processing on the output of the style encoder to obtain a post-processed style embedding, and then fusing the post-processed style embedding with the output of the audio-to-motion adapter. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images