A method and apparatus for speech-driven three-dimensional speaker animation generation

Abstract

The application discloses a speech-driven three-dimensional speaker animation generation method and device, and solves the problems of time jitter and style decoupling of a traditional continuous regression method by quantizing a continuous face motion parameter sequence into discrete tokens; a StyleEncoder is innovatively designed to extract a reference segment style vector, and the StyleEncoder is combined with a Prefilling strategy to be used as a prefix input of an autoregressive Transformer, so that natural style migration and controllable generation are realized; meanwhile, the audio features, motion tokens and style vectors are jointly modeled by using the Transformer, so that the animation and the speech rhythm are accurately aligned; finally, through high-fidelity decoding and FLAME2023 reconstruction rendering, three-dimensional speaker animation with consistent geometry, delicate expression and strong real-time performance is generated, and the generation stability, style controllability and visual reality are significantly improved, and the application is suitable for application scenarios such as virtual digital people and film post-production.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics, computer vision, speech signal processing, 3D face modeling and artificial intelligence, and in particular to a method and apparatus for generating 3D speaker animation driven by speech. Background Technology

[0002] Voice-driven 3D speaker animation aims to automatically generate 3D lip-sync, facial expression, and head movement sequences consistent with the speech content, pronunciation rhythm, and semantics based on the input speech. It represents a significant direction in current virtual digital human generation technology. Its main applications include virtual anchors, digital assistants, multimodal human-computer interaction, and lip-sync synchronization in film and television post-production. Current research focuses on three main methods: The first is action generation methods based on continuous regression, typically using neural networks such as LSTM, TCN, or Transformer to directly predict 3DMM or FLAME parameter sequences. While these methods are simple in structure and have fast inference speeds, they are prone to temporal jitter; furthermore, the mixing of action content and performance style in a continuous space makes style decoupling and transfer difficult. The second method is based on diffusion models, obtaining high-quality parameter sequences through progressive back-diffusion in a continuous action space. Although diffusion models excel in visual generation, their inference phase requires multiple iterations, resulting in significant latency, which is unfavorable for real-time applications; additionally, their ability to control style is limited. The third type of method is based on a discrete token-based action modeling strategy. It compresses continuous actions into a finite number of discrete action atoms using VQ-VAE, and then utilizes an autoregressive Transformer to model the action structure. This approach is highly effective in video, music, and audio generation, offering advantages such as high generation stability, clear action structure, and more flexible style control. However, current research has not yet incorporated the high expressive power of the FLAME2023 parametric model and lacks dedicated strategies for style transfer.

[0003] Compared to earlier versions, the FLAME2023 parametric model has been enhanced in terms of facial expression details, topological consistency, and dynamic structure, enabling it to more accurately represent changes in the 3D facial region. It is particularly suitable for building high-fidelity lip-sync and facial animations for digital humans.

[0004] Based on the review, existing methods mainly have the following limitations: it is difficult to achieve explicit decoupling of style and content in continuous action spaces; diffusion models have long inference times and are not suitable for real-time generation; they do not fully utilize the model capabilities of FLAME2023; they lack stable discrete action representations; and style control often relies on static vectors, which has limited effectiveness.

[0005] Existing voice-driven 3D speaker animation technologies suffer from several core problems. First, continuous regression-based methods directly map speech to FLAME or 3DMM parameter sequences, making it difficult to decouple facial movements from performance style, thus hindering style control or transfer during the generation stage. Second, continuous movement generation often suffers from temporal jitter, resulting in less smooth animation; while diffusion model-based methods generate higher quality, multi-step backdiffusion causes significant inference delays, which is unfavorable for real-time or near-real-time applications. Furthermore, existing methods do not fully utilize the high-fidelity expressive power of the FLAME2023 parameter model, resulting in insufficient facial detail and limitations in the realism and refinement of the generated animation. Finally, the lack of style transfer and controllable generation mechanisms makes it difficult to naturally transfer the movement style of reference segments to the target sequence. Summary of the Invention

[0006] To address existing problems, this invention proposes a voice-driven 3D speaker animation generation method. Continuous parameters are discretized into action tokens, and a style encoder and autoregressive Transformer are combined with a prefilling style transfer strategy to achieve style-controllable, high-fidelity, stable, and efficient voice-driven 3D speaker animation generation.

[0007] The second objective of this invention is to provide a voice-driven three-dimensional speaker animation generation device.

[0008] The third objective of this invention is to provide a computer device.

[0009] To achieve the above objectives, a first aspect of the present invention provides a method for generating a voice-driven three-dimensional speaker animation, comprising: Acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame, and preprocess the parameter sequence to output a standardized parameter sequence; A reference subsequence is extracted from the standardized parameter sequence, input into the style encoder to extract a multidimensional style vector, and the style encoder is trained by contrastive loss or triplet loss. The standardized parameter sequence is input into the VQ-VAE encoder to generate a latent vector. The vector is then quantized and mapped to the codebook to generate discrete tokens. Finally, the parameter sequence is reconstructed by the decoder. Tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors. After mapping action tokens, speech features, and style vectors, they are input into a multi-layer Transformer and autoregressive training is performed using causal masks. Extract the token sequence and style vector from the reference segment, and use the token sequence as a prefix to input the Transformer together with the test audio to generate the subsequent action token sequence through autoregression. The generated token sequence is decoded into normalized parameters, inversely normalized, and then input into the FLAME2023 model to reconstruct the facial mesh. Combined with camera parameters, the final animation video is rendered and output.

[0010] In one embodiment of the present invention, video frames, synchronized audio, and camera parameters are received; a single-frame reconstruction network is used to estimate the FLAME2023 parameter vector for each frame; and the parameter sequence is preprocessed to output a standardized parameter sequence, including: Acquire multimodal camera data including video frame sequences, synchronized audio, and camera parameters; The VHAP or equivalent single-frame reconstruction network is used to estimate each frame of the multimodal camera data, generating a FLAME2023 parameter vector sequence consisting of identity, expression, pose and camera parameters; The FLAME2023 parameter vector sequence is time-aligned, pruned or padded to a fixed length, and normalized based on global statistics to obtain a standardized FLAME2023 parameter sequence.

[0011] In one embodiment of the present invention, a reference subsequence is extracted from a standardized parameter sequence, input into a style encoder to extract a multidimensional style vector, and the style encoder is trained by contrastive loss or triplet loss, including: A reference subsequence is extracted from the standardized FLAME2023 parameter sequence and input into the StyleEncoder network to extract a 128-dimensional style vector representing the performance style. StyleEncoder sequentially extracts and maps features through a structure consisting of three layers of one-dimensional convolution, positional encoding, two layers of Transformer encoder, temporal average pooling, and fully connected layers. Training is constrained by contrastive loss or triplet loss to bring style vectors of samples with the same identity / style closer together and keep style vectors of samples with different styles separate.

[0012] In one embodiment of the present invention, a standardized parameter sequence is input into a VQ-VAE encoder to generate a latent vector, which is then mapped to a codebook via vector quantization to generate discrete tokens. The parameter sequence is then reconstructed by a decoder, including: The standardized FLAME2023 parameter sequence is input into the VQ-VAE encoder to generate a 128-dimensional latent vector; Discrete action tokens are generated by mapping vector quantization to a codebook containing 512 code vectors, and then reconstructed into the original parameter dimension by a symmetrical four-layer one-dimensional convolutional decoder, thus realizing the discretization representation of continuous action parameters. The training loss is a weighted sum of reconstruction loss, codebook update loss, and submission loss. The stop-gradient operation is used to update the codebook and encoder respectively, constraining the discrete tokens to reconstruct the original parameter sequence with high fidelity.

[0013] In one embodiment of the present invention, the standardized parameter sequences of all training samples are tokenized, and speech features and style vectors of reference segments are extracted simultaneously to construct a training dataset containing action tokens, speech features, and style vectors, including: VQ-VAE is applied to the standardized FLAME2023 parameter sequence of all training samples to generate discrete action token sequences; Frame-by-frame speech features are extracted from synchronized audio using a Hubert-based or equivalent audio encoder, and a StyleEncoder is applied to a reference segment for each sample to generate a style vector. The action token sequence, speech feature sequence, and style vector are combined to construct a dataset for training the autoregressive model.

[0014] In one embodiment of the present invention, the action token, speech features, and style vector are mapped and input into a multi-layer Transformer, and autoregressive training is performed using causal masks, including: After mapping the action token sequence, speech features, and style vectors to a unified dimension, they are input together with the positional encoding into a multi-layer Transformer model using causal masking for feature fusion. During training, the token sequence with the start symbol prefix is ​​used as input, and the target token sequence is predicted by autoregression to maximize the conditional probability of the current token based on historical tokens, audio features and style vectors; The model learns discrete action token generation driven by the joint speech rhythm and performance style, enabling facial animation sequence modeling that is synchronized with speech and style controllable.

[0015] In one embodiment of the present invention, a token sequence and style vector are extracted from a reference segment. The token sequence is used as a prefix and input into a Transformer along with the test audio. An autoregressive generation of subsequent action token sequences is performed, including: During the inference phase, the standardized FLAME2023 parameter sequence of the reference fragment is tokenized by VQ-VAE to obtain a reference token sequence, and the reference style vector is extracted by StyleEncoder; The reference token sequence is used as a prefix and input together with the test audio features into the trained autoregressive Transformer model to predict subsequent action tokens frame by frame under the reference style vector condition. Generate a complete sequence containing a reference prefix and a predicted token to achieve target animation compositing guided by the reference fragment style.

[0016] In one embodiment of the present invention, the generated token sequence is decoded into normalized parameters, inverse normalized, and then input into the FLAME2023 model to reconstruct the facial mesh. The final animated video is then rendered and output based on camera parameters, including: The generated action token sequence is input into the VQ-VAE decoder to obtain the normalized parameter sequence. The statistical mean and standard deviation calculated during the training phase are used to perform inverse normalization to restore the FLAME2023 parameter sequence of the original parameter space. The inversely normalized parameters are input frame by frame into the FLAME2023 model to reconstruct the vertex coordinates and fixed triangular patch topology of the facial geometry mesh, generating a three-dimensional facial geometry sequence. Vertex transformation, projection, rasterization, and shading rendering are performed on the facial geometry sequence. A frame-by-frame image sequence is generated by combining camera parameters and lighting parameters, and then synthesized with the original audio to form the final 3D speaker animation video.

[0017] To achieve the above objectives, a second aspect of this application provides a voice-driven three-dimensional speaker animation generation apparatus, comprising: The parameter acquisition module is used to acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame, and preprocess the parameter sequence to output a standardized parameter sequence. The encoder training module is used to extract a reference subsequence from the standardized parameter sequence, input it into the style encoder to extract a multidimensional style vector, and train the style encoder by contrastive loss or triplet loss. The action discretization module is used to input the standardized parameter sequence into the VQ-VAE encoder to generate a latent vector, which is then mapped to the codebook through vector quantization to generate a discrete token, and finally the parameter sequence is reconstructed by the decoder. The dataset construction module is used to tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors. The autoregressive training module is used to map action tokens, speech features and style vectors and input them into a multi-layer Transformer, and use causal masks for autoregressive training. The inference output module is used to extract the token sequence and style vector from the reference segment. The token sequence is used as a prefix and input into the Transformer along with the test audio. The subsequent action token sequence is generated through autoregression. The decoding and rendering module is used to decode the generated token sequence into normalized parameters, inverse normalize them, input them into the FLAME2023 model to reconstruct the facial mesh, and combine them with camera parameters to render and output the final animation video.

[0018] To achieve the above objectives, a third aspect of the present invention provides an electronic device comprising: processor; A memory that stores executable instructions; when the processor executes the instructions, it implements the method as claimed in any one of claims 1-8.

[0019] The methods, apparatus, and computer devices of this invention realize explicit decoupling and controllable transfer of facial action content and performance style in voice-driven 3D speaker animation, thereby improving the stability, style consistency, and high-fidelity reproduction of facial expression details in the generated animation.

[0020] The technical effects of this invention are as follows: This invention achieves high-fidelity generation of speech-driven 3D speaker animation by combining FLAME2023 parametric representation, VQ-VAE discrete action encoding, autoregressive Transformer, and prefilling style transfer strategies. Compared with existing technologies, this invention has many beneficial effects.

[0021] First, discrete action coding can effectively stabilize action sequences, significantly reduce jitter and distortion that occur in traditional continuous regression methods, while maintaining the naturalness of facial expression details and mouth shape changes.

[0022] Secondly, the style encoder and prefilling strategy enable the performance style of the reference segment to be accurately transferred to the generated sequence, enhancing the style consistency and controllability of the animation and meeting the needs of diverse digital human applications. Furthermore, by using an autoregressive Transformer to jointly model audio features, action tokens, and style vectors, the generated animation is highly aligned with the speech rhythm, improving the synchronization and naturalness of speech-driven facial movements.

[0023] Finally, by combining the high-fidelity FLAME2023 parametric reconstruction and rendering workflow, this invention can output geometrically consistent, expressive, and visually realistic 3D animations, providing a directly usable high-quality solution for virtual digital humans, film and television post-production, and intelligent interactive systems. These technical advantages not only improve the stability and quality of animation generation but also provide new methods and theoretical support for the research and application of style-controllable voice-driven 3D facial animation. Attached Figure Description

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a voice-driven three-dimensional speaker animation generation method provided in an embodiment of the present invention; Figure 2 An architecture diagram of a voice-driven 3D speaker animation generation method provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a voice-driven three-dimensional speaker animation generation device provided in an embodiment of the present invention; Figure 4 The computer device provided in the embodiments of the present invention. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] The following description, with reference to the accompanying drawings, describes a method, apparatus, and computer device for generating voice-driven three-dimensional speaker animation according to embodiments of the present invention.

[0028] This embodiment provides a method for generating 3D speaker animation driven by voice. For example... Figure 1 and Figure 2 As shown, the method includes the following steps: S1: Acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame, and preprocess the parameter sequence to output a standardized parameter sequence.

[0029] In one embodiment of the present invention, step S1 is input data preparation and FLAME2023 parameter reconstruction: including acquiring multimodal camera data containing video frame sequences, synchronized audio, and camera parameters; estimating each frame of the multimodal camera data using VHAP or an equivalent single-frame reconstruction network to generate a FLAME2023 parameter vector sequence composed of identity, expression, pose, and camera parameters; performing temporal alignment, fixed-length cropping or padding on the FLAME2023 parameter vector sequence, and performing normalization processing based on global statistics to obtain a standardized FLAME2023 parameter sequence.

[0030] Specifically, the input data preparation and FLAME2023 parameter reconstruction includes the following sub-steps: S10, Input data format: Each training / test sample contains a sequence of video frames. (Sampling rate 25 fps), synchronized audio (Sampling rate 16 kHz), and optional camera parameters. .

[0031] S11, Estimating FLAME2023 parameters using VHAP: for each frame. The FLAME2023 parameter vector is estimated using VHAP or an equivalent single-frame reconstruction network. , recorded as

[0032] in For the identity (shape) parameter (shared throughout the segment), For facial expression parameters, These are the head and jaw posture parameters. These are the camera parameters. From this, we obtain the parameter sequence. .

[0033] S12, Sequence Alignment and Cropping: Align audio frames to match the video frame rate; align parameter sequences to a fixed length. (e.g., 64 frames) perform cropping or padding; calculate the global mean. with standard deviation (Or one-dimensional variance) and perform normalization to obtain standardized parameters:

[0034] in This is a numerically stable term. The output is a normalized FLAME2023 parameter sequence. With synchronized audio features.

[0035] Step S1 uses the FLAME2023 parametric model to extract identity, expression, pose, and camera parameters, which can more accurately express subtle changes in the 3D face region, providing a strong expressive foundation for high-fidelity animation generation. By performing strict temporal alignment, fixed-length cropping, and global statistical normalization on the audio and video sequences, data scale differences and temporal inconsistencies are effectively eliminated, establishing a stable and standardized input foundation for subsequent VQ-VAE discrete coding and autoregressive modeling. This preprocessing process ensures the numerical stability and spatiotemporal continuity of the parameter sequences, significantly improving the convergence speed and generation quality of subsequent model training.

[0036] S2 extracts a reference subsequence from the standardized parameter sequence, inputs it into the style encoder to extract a multidimensional style vector, and trains the style encoder by contrastive loss or triplet loss.

[0037] In one embodiment of the present invention, step S2 is the training of the StyleEncoder, which includes: extracting reference subsequences from the standardized FLAME2023 parameter sequence and inputting them into the StyleEncoder network to extract a 128-dimensional style vector representing the performance style; the StyleEncoder sequentially performs feature extraction and mapping through a structure of three layers of one-dimensional convolution, positional encoding, two layers of Transformer encoder, and temporal average pooling and fully connected layers; and training is constrained by contrastive loss or triplet loss to make the style vectors of samples with the same identity / style close to each other and keep the style vectors of samples with different styles separate.

[0038] Specifically, training the StyleEncoder includes the following sub-steps: S20: Reference segment selection: Select a segment of length from each sequence. (e.g., 32 frames) reference subsequence As a style example.

[0039] S21: StyleEncoder Network Structure: StyleEncoder uses time series... The input undergoes three layers of Conv1d encoding (dimensions vary). Mapped to 128, 256, 256), each layer incorporates LayerNorm, ReLU, and Dropout, followed by positional encoding and two Transformer Encoder layers (each layer has 4 attention heads, hidden=256, FFN structure 256→512→256), then time-averaged pooling and two fully connected layers to output a 128-dimensional style vector.

[0040] S22: Style Constraint Training: To make style vectors discriminative and consistent, contrastive loss or triplet loss can be used to constrain style vectors of samples with the same identity / style to be close to each other, while maintaining distance between different styles.

[0041] In step S2, the StyleEncoder extracts a 128-dimensional style vector from the reference segment using a convolutional and Transformer encoder structure. This vector is then trained with contrastive loss or triplet loss constraints, resulting in style vectors with strong discriminative power and consistency, accurately representing and distinguishing different performance styles. This encoding mechanism achieves explicit decoupling of style information from speech content and facial movements, providing independent style conditional inputs for subsequent controllable style generation. The extracted stable style vectors serve as high-level semantic representations, providing precise style guidance for the Prefilling strategy and autoregressive Transformer generation, significantly improving the continuity and naturalness of style transfer. S3 inputs the standardized parameter sequence into the VQ-VAE encoder to generate a latent vector, which is then mapped to the codebook through vector quantization to generate a discrete token, and finally the parameter sequence is reconstructed by the decoder.

[0042] In one embodiment of the present invention, step S3 is the action discretization of the FLAME2023 parameter model based on VQ-VAE, including: inputting the standardized FLAME2023 parameter sequence into the VQ-VAE encoder to generate a 128-dimensional latent vector; generating discrete action tokens by mapping the vectors to a codebook containing 512 code vectors, and then reconstructing them into the original parameter dimensions by a symmetrical four-layer one-dimensional convolutional decoder, thereby realizing the discretization representation of continuous action parameters; the training loss is composed of a weighted sum of reconstruction loss, codebook update loss, and submission loss, and the stop-gradient operation is used to update the codebook and encoder respectively, constraining the discrete tokens to reconstruct the original parameter sequence with high fidelity.

[0043] Specifically, the action discretization of the FLAME2023 parametric model based on VQ-VAE includes the following sub-steps: S30, Encoder structure: This structure normalizes the parameter sequence. The signal is fed into a four-layer Conv1d encoder, and the channel changes are as follows: To obtain the potential vector sequence .

[0044] S31, Vector Quantization (Codebook): Setting the codebook (For example Each vector has a dimension of 128. For each latent vector Perform nearest neighbor quantization: This yields a discrete token sequence. .

[0045] S32, Decoder structure: Employs a symmetrical Conv1d decoder (128→256→256→256→ The quantized latent vector sequence Decoded into normalized parameters for reconstruction .

[0046] S33, Training Loss: The training objective consists of reconstruction loss, codebook update loss, and commit loss.

[0047] The total loss is:

[0048] in This indicates a stop-gradient operation. These are hyperparameters. The trained VQ-VAE (including Encoder, Codebook, and Decoder) can map continuous FLAME2023 parameter sequences to discrete action tokens and reconstruct high-fidelity parameter sequences.

[0049] Step S3 discretizes the continuous FLAME2023 parameter sequence into a finite number of action tokens using VQ-VAE, effectively eliminating temporal jitter and accumulated errors in traditional continuous regression methods, and achieving stable representation and structural decoupling of facial actions. Vector quantization combined with a symmetric decoder architecture can maintain high-fidelity reconstruction capabilities while compressing the action space, accurately restoring facial expression details and lip shape changes. The discretized representation provides a concise and efficient sequence of action atoms for subsequent autoregressive Transformer modeling, greatly improving the stability, controllability, and computational efficiency of the generation process, and is a key foundation for achieving style transfer and high-quality animation compositing.

[0050] S4. Tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors.

[0051] In one embodiment of the present invention, step S4 is to construct a token dataset, including applying VQ-VAE to the normalized FLAME2023 parameter sequences of all training samples to generate discrete action token sequences; extracting frame-by-frame speech features from synchronized audio using a HuBERT-base or equivalent audio encoder, and applying StyleEncoder to the reference segment of each sample to generate style vectors; and combining the action token sequences, speech feature sequences, and style vectors to construct a dataset for training an autoregressive model.

[0052] Specifically, the construction of the Token dataset includes the following sub-steps: S40, Tokenization: Apply VQ-VAE to all training data: Generate a token sequence.

[0053] S41, Speech Feature Extraction: Extract frame-by-frame speech embeddings (e.g., 768-dimensional) from the raw audio using a Hubert-based or equivalent audio encoder, and align them to the video frame duration. .

[0054] S42, Generate style vectors: Apply StyleEncoder to the reference fragment for each sample to obtain the style vector. .

[0055] S43, Save training samples: Each training sample contains three parts: action token sequence. The corresponding speech feature sequence and reference style vectors Organize them into a dataset for training autoregressive models.

[0056] This step involves uniformly tokenizing all training samples using VQ-VAE, extracting HuberT speech features, and generating style vectors using StyleEncoder. This constructs a multimodal aligned dataset containing action tokens, audio features, and style vectors, ensuring precise temporal and semantic matching of the three heterogeneous datasets. The combined representation of discrete tokens and continuous features significantly reduces the input complexity and computational cost of subsequent autoregressive models, improving training efficiency and convergence stability. Explicitly separated style representations provide supervisory signals for the model to learn independent style control mechanisms, enabling flexible style transfer and controllable synthesis during the generation stage, laying the foundation for decoupled style generation.

[0057] S5 maps action tokens, speech features, and style vectors and inputs them into a multi-layer Transformer, then uses causal masks for autoregressive training.

[0058] In one embodiment of the present invention, step S5 is autoregressive Transformer (LM-Transformer) training, including: mapping the action token sequence, speech features and style vectors to a unified dimension, and inputting them together with the positional encoding into a multi-layer Transformer model using causal masking for feature fusion; during training, the token sequence with the start symbol prefix is ​​used as input, and the target token sequence is predicted autoregressively to maximize the conditional probability of the current token based on historical tokens, audio features and style vectors; the model learns to generate discrete action tokens jointly driven by speech rhythm and performance style, thereby realizing facial animation sequence modeling that is synchronized with speech and has controllable style.

[0059] Specifically, the training of this autoregressive Transformer (LM-Transformer) includes the following sub-steps: S50, Model Structure: The Transformer input includes token embedding (mapping token IDs to 512 dimensions), positional encoding, audio feature mapping (mapping 768 dimensions to 512 dimensions), and style mapping (mapping 128 dimensions to 512 dimensions). The backbone is a multi-layer (e.g., 8-layer) Transformer, employing 8-head attention, FFN 512→2048→512, and using a causal mask during autoregressive training. The output layer maps the hidden dimension to the vocabulary size (vocab_size).

[0060] S51, Training Objective: During training, the input sequence is in the form of... The goal is Maximize the autoregressive conditional probability:

[0061] The model learns the speech-driven discrete action generation distribution by jointly modeling action tokens, audio conditions, and style conditions.

[0062] This step uses an autoregressive Transformer to jointly model audio features, action tokens, and style vectors, capturing long-term temporal dependencies and achieving precise alignment and semantic association between speech rhythm and facial movements. The causal masking mechanism ensures that the generation process strictly follows temporal causality, avoiding interference from future information and making the action sequence coherent and natural in the temporal domain, significantly reducing jitter and distortion. This training paradigm gives the model flexible zero-shot style control capabilities, and diverse performance style transfer can be achieved by simply replacing the style vectors during the inference stage, greatly improving the controllability and applicability of the generated animation.

[0063] S6 extracts the token sequence and style vector from the reference segment, uses the token sequence as a prefix and inputs it into the Transformer along with the test audio, and generates the subsequent action token sequence through autoregression.

[0064] In one embodiment of the present invention, step S6 is Prefilling style transfer inference, which includes extracting a token sequence and style vector from the reference segment, inputting the token sequence as a prefix and the test audio together into the Transformer, and generating a subsequent action token sequence through autoregression. This includes: in the inference stage, performing VQ-VAE tokenization on the normalized FLAME2023 parameter sequence of the reference segment to obtain a reference token sequence, and extracting a reference style vector through StyleEncoder; inputting the reference token sequence as a prefix and the test audio features together into the trained autoregressive Transformer model, and predicting subsequent action tokens frame by frame under the condition of the reference style vector; generating a complete sequence containing the reference prefix and the predicted tokens, thereby realizing the synthesis of the target animation guided by the style of the reference segment.

[0065] Specifically, Prefilling style transfer inference includes the following sub-steps: S60, Reference Fragment Token Extraction: During the inference phase, a reference fragment is selected. And obtain the corresponding token sequence through VQ-VAE. .

[0066] S61, Prefilling: Prefilling As a prefix input to the Transformer, and then in audio conditions With reference style vector Under the combined effect of these factors, subsequent tokens are generated through autoregression. For the first... Frame, have

[0067] S62, Output: Generate a complete token sequence. This enables the synthesis of action sequences guided by the style of reference segments.

[0068] The Prefilling strategy in this step uses the reference fragment token as a prefix and inputs it together with the style vector, allowing the generated sequence to naturally inherit the reference performance style, achieving precise style transfer and style consistency control. The reference token provides the initial action pattern and style base, ensuring the continuity and smooth transition of style expression in the autoregressive generation process, effectively avoiding style abruptness or mismatch problems. This mechanism supports flexibly replacing the reference fragment during the inference stage to change the generated style without retraining the model, significantly improving the system's style controllability and diversified application capabilities.

[0069] S7 decodes the generated token sequence into normalized parameters, inverse normalizes them, inputs them into the FLAME2023 model to reconstruct the facial mesh, and combines them with camera parameters to render and output the final animation video.

[0070] In one embodiment of the present invention, step S7 is token decoding and FLAME2023 rendering, including inputting the generated action token sequence into the VQ-VAE decoder to obtain a normalized parameter sequence, performing inverse normalization using the statistical mean and standard deviation calculated during the training phase to restore the FLAME2023 parameter sequence in the original parameter space; inputting the inverse normalized parameters frame by frame into the FLAME2023 model to reconstruct the vertex coordinates and fixed triangular facet topology of the facial geometric mesh to generate a three-dimensional facial geometric sequence; performing vertex transformation, projection, rasterization and shading rendering on the facial geometric sequence, combining camera parameters and lighting parameters to generate a frame-by-frame image sequence, and synchronously synthesizing it with the original audio to form the final three-dimensional speaker animation video.

[0071] Specifically, token decoding and FLAME2023 rendering include the following sub-steps: S70, Token → Continuous FLAME2023 Parameter: The generated token sequence Inputting the VQ-VAE decoder yields the reconstructed normalized parameter sequence. Then, inverse normalize the statistics from the training phase:

[0072] in The mean and standard deviation (dimension-by-dimension or global) are calculated in S12. This indicates multiplication by dimension (broadcast).

[0073] S71, FLAME2023 Mesh Reconstruction: Using the official FLAME2023 model files (including face base model, skeleton, and skin weights), the parameters are reconstructed frame by frame. Input FLAME2023 to get the vertex coordinates: Patch topology For a fixed set of triangular facets.

[0074] S72, Rendering Process Description: The rendering module receives vertices. noodle pieces Material / Vertex Color Camera parameters This involves lighting parameters (ambient light and directional light intensity, etc.). The rendering process includes vertex transformation, projection, rasterization, and shading. Pixel color can be represented as:

[0075] in For ambient light intensity, and The first The intensity and direction of each light source This represents the surface normal vector. The resulting image sequence is obtained after frame-by-frame rendering. , with the original audio The final 3D speaker animation video is synthesized from the original video.

[0076] This step uses VQ-VAE decoding combined with statistical inverse normalization to accurately map discrete tokens to continuous FLAME2023 parameters, achieving a complete and precise reconstruction from discrete representation to facial geometry. Leveraging the powerful parametric capabilities of the FLAME2023 model, it generates a sequence of 3D facial meshes with topological consistency and rich detail, ensuring geometric realism and expression accuracy. Finally, it combines camera and lighting parameters for rendering and synchronizes with audio, outputting a high-quality 3D speaker animation video that can be directly used in applications such as virtual digital humans and film post-production, forming an end-to-end closed loop from data input to video output.

[0077] In summary, the voice-driven 3D speaker animation generation method of the present invention can achieve the following effects: (1) A discrete action encoding method based on FLAME2023 parametric representation is proposed. This method quantizes the continuous facial action parameter sequence into discrete action tokens through VQ-VAE, realizing stable representation and high-fidelity reconstruction of actions. This design breaks through the limitations of traditional continuous regression methods in terms of generation stability and style control, making action sequences easier to style transfer and autoregressive modeling while maintaining details.

[0078] (2) A style encoder and a prefilling style transfer strategy were designed. The StyleEncoder can extract performance style vectors from the reference segment, and the prefilling strategy uses the reference style as a prefix input to the Transformer to guide the autoregressive generation of action sequences. This mechanism ensures that the generated sequence can naturally inherit the style of the reference segment, achieving style continuity and controllability, and solving the problem of unstable style transfer effect in traditional methods.

[0079] (3) An autoregressive Transformer model for audio conditional fusion is proposed. This model innovatively models audio features, action tokens and style vectors in a multi-layer Transformer, so that the generated action sequence is strictly aligned with the speech rhythm, while supporting flexible facial performance style control, which significantly improves the accuracy and naturalness of speech-driven 3D animation generation.

[0080] (4) A high-fidelity FLAME2023 parameter reconstruction and rendering workflow is proposed. The generated discrete tokens are decoded into continuous parameters, and inverse normalization is performed in combination with the statistical normalization in the training phase. These parameters are then input into the FLAME2023 model to achieve high-precision facial geometry and expression reconstruction. This workflow ensures the superiority of the rendered animation in terms of geometric consistency, expression details, and visual realism, providing a directly applicable solution for virtual digital humans, film and television post-production, and intelligent interaction.

[0081] like Figure 3 As shown, the present invention proposes a voice-driven three-dimensional speaker animation generation device 10, comprising: The parameter acquisition module 100 is used to acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector of each frame, and preprocess the parameter sequence to output a standardized parameter sequence. The encoder training module 200 is used to extract a reference subsequence from the standardized parameter sequence, input it into the style encoder to extract a multidimensional style vector, and train the style encoder by contrastive loss or triplet loss. The action discretization module 300 is used to input the standardized parameter sequence into the VQ-VAE encoder to generate a latent vector, and then to generate a discrete token by mapping the vector to the codebook through vector quantization, and finally to reconstruct the parameter sequence by the decoder. The dataset construction module 400 is used to tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features and style vectors. The autoregressive training module 500 is used to map action tokens, speech features and style vectors and input them into a multi-layer Transformer, and perform autoregressive training using causal masks. The inference output module 600 is used to extract the token sequence and style vector from the reference segment, and input the token sequence as a prefix and the test audio into the Transformer to generate the subsequent action token sequence through autoregression. The decoding and rendering module 700 is used to decode the generated token sequence into normalized parameters, inverse normalize them, input them into the FLAME2023 model to reconstruct the facial mesh, and combine them with camera parameters to render and output the final animation video.

[0082] The voice-driven 3D speaker animation generation device of this invention realizes explicit decoupling and controllable transfer of facial action content and performance style in voice-driven 3D speaker animation, thereby improving the stability, style consistency and high-fidelity reproduction of facial expression details in the generated animation.

[0083] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 4 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for generating 3D speaker animation driven by voice, characterized in that, include: Acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame, and preprocess the parameter sequence to output a standardized parameter sequence; A reference subsequence is extracted from the standardized parameter sequence, input into the style encoder to extract a multidimensional style vector, and the style encoder is trained by contrastive loss or triplet loss. The standardized parameter sequence is input into the VQ-VAE encoder to generate a latent vector. The vector is then quantized and mapped to the codebook to generate discrete tokens. Finally, the parameter sequence is reconstructed by the decoder. Tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors. After mapping action tokens, speech features, and style vectors, they are input into a multi-layer Transformer and autoregressive training is performed using causal masks. Extract the token sequence and style vector from the reference segment, and use the token sequence as a prefix to input the Transformer together with the test audio to generate the subsequent action token sequence through autoregression. The generated token sequence is decoded into normalized parameters, inversely normalized, and then input into the FLAME2023 model to reconstruct the facial mesh. Combined with camera parameters, the final animation video is rendered and output.

2. The method as described in claim 1, characterized in that, The system receives video frames, synchronized audio, and camera parameters. It uses a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame and preprocesses the parameter sequence to output a standardized parameter sequence, including: Acquire multimodal camera data including video frame sequences, synchronized audio, and camera parameters; The VHAP or equivalent single-frame reconstruction network is used to estimate each frame of the multimodal camera data, generating a FLAME2023 parameter vector sequence consisting of identity, expression, pose and camera parameters; The FLAME2023 parameter vector sequence is time-aligned, pruned or padded to a fixed length, and normalized based on global statistics to obtain a standardized FLAME2023 parameter sequence.

3. The method as described in claim 2, characterized in that, A reference subsequence is extracted from the standardized parameter sequence and input into the style encoder to extract a multidimensional style vector. The style encoder is then trained using contrastive loss or triplet loss, including: A reference subsequence is extracted from the standardized FLAME2023 parameter sequence and input into the StyleEncoder network to extract a 128-dimensional style vector representing the performance style. StyleEncoder sequentially extracts and maps features through a structure consisting of three layers of one-dimensional convolution, positional encoding, two layers of Transformer encoder, temporal average pooling, and fully connected layers. Training is constrained by contrastive loss or triplet loss to bring style vectors of samples with the same identity / style closer together and keep style vectors of samples with different styles separate.

4. The method as described in claim 3, characterized in that, The standardized parameter sequence is input into the VQ-VAE encoder to generate a latent vector. This vector is then quantized and mapped to a codebook to generate discrete tokens. Finally, the decoder reconstructs the parameter sequence, including: The standardized FLAME2023 parameter sequence is input into the VQ-VAE encoder to generate a 128-dimensional latent vector; Discrete action tokens are generated by mapping vector quantization to a codebook containing 512 code vectors, and then reconstructed into the original parameter dimension by a symmetrical four-layer one-dimensional convolutional decoder, thus realizing the discretization representation of continuous action parameters. The training loss is composed of a weighted sum of reconstruction loss, codebook update loss, and submission loss. The stop-gradient operation is used to update the codebook and encoder respectively, constraining the discrete tokens to reconstruct the original parameter sequence with high fidelity.

5. The method as described in claim 4, characterized in that, Tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors, including: VQ-VAE is applied to the standardized FLAME2023 parameter sequence of all training samples to generate discrete action token sequences; Frame-by-frame speech features are extracted from synchronized audio using a Hubert-based or equivalent audio encoder, and a StyleEncoder is applied to a reference segment for each sample to generate a style vector. The action token sequence, speech feature sequence, and style vector are combined to construct a dataset for training the autoregressive model.

6. The method as described in claim 5, characterized in that, After mapping action tokens, speech features, and style vectors, the data is input into a multi-layer Transformer and trained using causal masks for autoregression, including: After mapping the action token sequence, speech features, and style vectors to a unified dimension, they are input together with the positional encoding into a multi-layer Transformer model using causal masking for feature fusion. During training, the token sequence with the start symbol prefix is ​​used as input, and the target token sequence is predicted by autoregression to maximize the conditional probability of the current token based on historical tokens, audio features and style vectors; The model learns discrete action token generation driven by the joint speech rhythm and performance style, enabling facial animation sequence modeling that is synchronized with speech and style controllable.

7. The method as described in claim 6, characterized in that, Extract the token sequence and style vector from the reference segment. Use the token sequence as a prefix and input it along with the test audio into the Transformer. Generate the subsequent action token sequence through autoregression, including: During the inference phase, the standardized FLAME2023 parameter sequence of the reference fragment is tokenized by VQ-VAE to obtain a reference token sequence, and the reference style vector is extracted by StyleEncoder; The reference token sequence is used as a prefix and input together with the test audio features into the trained autoregressive Transformer model to predict subsequent action tokens frame by frame under the reference style vector condition. Generate a complete sequence containing a reference prefix and a predicted token to achieve target animation compositing guided by the reference fragment style.

8. The method as described in claim 7, characterized in that, The generated token sequence is decoded into normalized parameters, inverse normalized, and then input into the FLAME2023 model to reconstruct the facial mesh. Combined with camera parameters, the final animated video is rendered and output, including: The generated action token sequence is input into the VQ-VAE decoder to obtain the normalized parameter sequence. The statistical mean and standard deviation calculated during the training phase are used to perform inverse normalization to restore the FLAME2023 parameter sequence of the original parameter space. The inversely normalized parameters are input frame by frame into the FLAME2023 model to reconstruct the vertex coordinates and fixed triangular patch topology of the facial geometry mesh, generating a three-dimensional facial geometry sequence. Vertex transformation, projection, rasterization, and shading rendering are performed on the facial geometry sequence. A frame-by-frame image sequence is generated by combining camera parameters and lighting parameters, and then synthesized with the original audio to form the final 3D speaker animation video.

9. A voice-driven 3D speaker animation generation device, characterized in that, include: The parameter acquisition module is used to acquire multimodal camera data, use a single-frame reconstruction network to estimate the FLAME2023 parameter vector for each frame, and preprocess the parameter sequence to output a standardized parameter sequence. The encoder training module is used to extract a reference subsequence from the standardized parameter sequence, input it into the style encoder to extract a multidimensional style vector, and train the style encoder by contrastive loss or triplet loss. The action discretization module is used to input the standardized parameter sequence into the VQ-VAE encoder to generate a latent vector, which is then mapped to the codebook through vector quantization to generate a discrete token, and finally the parameter sequence is reconstructed by the decoder. The dataset construction module is used to tokenize the standardized parameter sequences of all training samples, simultaneously extract speech features and style vectors of reference segments, and construct a training dataset containing action tokens, speech features, and style vectors. The autoregressive training module is used to map action tokens, speech features and style vectors and input them into a multi-layer Transformer, and use causal masks for autoregressive training. The inference output module is used to extract the token sequence and style vector from the reference segment. The token sequence is used as a prefix and input into the Transformer along with the test audio. The subsequent action token sequence is generated through autoregression. The decoding and rendering module is used to decode the generated token sequence into normalized parameters, inverse normalize them, input them into the FLAME2023 model to reconstruct the facial mesh, and combine them with camera parameters to render and output the final animation video.

10. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a voice-driven three-dimensional speaker animation generation method as described in any one of claims 1-8.

Images