Three-dimensional face animation generation method and device, equipment and medium

Abstract

The invention provides a three-dimensional face animation generation method and device, equipment and a medium, and relates to the field of animation generation. The method comprises the following steps: acquiring a human face motion sequence and a voice signal, and capturing human face motion global features and human face motion local features through a double-layer face encoder; generating face global motion prediction through an upper layer decoder, and fusing the face motion local features to obtain face motion features; voice content features and emotion features of the voice signals are obtained through a voice content encoder and a voice emotion encoder, and style features are obtained through one-hot encoding; performing feature enhancement on the emotion features based on a bidirectional long-short term memory network to obtain emotion time-frequency enhancement features, and fusing the emotion time-frequency enhancement features with the voice content features and the style features to obtain voice features; and inputting the human face motion features and the voice features into a lower face decoder to obtain human face motion prediction, and generating a three-dimensional human face animation. According to the invention, more accurate, natural and diversified three-dimensional face animation generation can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of animation generation and computer graphics, and in particular to a method, device, equipment and medium for generating three-dimensional facial animation. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, computer graphics, and speech processing, 3D virtual human technology, as a core vehicle for the next generation of human-computer interaction, is gradually moving from theoretical research to practical application. Currently, this technology has been widely applied in fields such as intelligent customer service, virtual live streaming, and smart cities. Speech-driven 3D facial animation generation, a core research direction, significantly improves the naturalness of expression and the credibility of interaction with 3D virtual humans by converting speech signals into synchronized facial animations, which is of great significance for optimizing the human-computer interaction experience. However, while current speech-driven 3D facial animation generation methods have made significant progress in generating high-quality animations at low cost, most methods focus primarily on lip synchronization and personal style control, neglecting facial emotional expression. This makes it difficult for the generated animations to accurately convey the emotional information contained in the speech content, thus limiting their expressiveness and user experience in practical applications.

[0003] Previous research has primarily focused on expressing emotional features in speech through the introduction of emotion labels or expression estimation. While the introduction of emotion labels can enhance expressiveness to a certain extent, its results still have limitations. Expression estimation methods, while able to improve the diversity of 3D facial animations, still lack generalization capabilities. Recently, Emotalk has made progress by using a cross-reconstruction loss to decouple speech content from emotion and employing an end-to-end neural network to generate emotionally controllable 3D facial animations. However, due to its autoregressive approach, the output at each time step is completely dependent on the previous frame, consistently generating consistent results for the same audio input, lacking diversity. This generation approach can lead to relatively rigid 3D facial animations, hindering the ability to exhibit rich personalization and randomness. In contrast, ProbTalk3D, a 3D face generation method based on VQ-VAE, introduces vector quantization modeling to improve the diversity and naturalness of facial animations. It can generate different and plausible expressions from the same speech input, overcoming the problem of previous methods that tend to generate a single expression. However, ProbTalk3D fails to effectively distinguish between the content features and emotional features in speech. In the case of the same speech content but different emotions, the generated facial expressions are easily confused and cannot accurately correspond to different emotions.

[0004] To address the above issues, there is an urgent need for a technical solution that can deeply explore the complex connection between speech emotion characteristics and three-dimensional facial animation to improve the accuracy and diversity of generated animations.

[0005] In view of this, this application is hereby filed. Summary of the Invention

[0006] The present invention aims to provide a method, device, equipment and medium for generating three-dimensional facial animation to address the shortcomings of existing methods in terms of emotional expression and diversity, and to achieve more accurate, natural and diverse three-dimensional facial animation generation.

[0007] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: A method for generating three-dimensional facial animation, comprising: S1, obtains facial motion sequence and speech signal; S2, according to the facial motion sequence, capturing global facial motion features and local facial motion features respectively through a dual-layer facial encoder; the dual-layer facial encoder includes an upper-layer facial encoder and a lower-layer facial encoder, each encoder including a convolutional layer, a linear projection layer, and a multi-layer Transformer layer; S3, generating a global face motion prediction through an upper-layer facial decoder based on the global face motion feature, and fusing it with the local face motion feature to obtain a face motion feature; S4, passing the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtaining style features through one-hot encoding; S5, performing feature enhancement on the emotional feature based on a bidirectional long short-term memory network to obtain an emotional time-frequency enhancement feature, and performing multi-scale fusion and encoding with the speech content feature and the style feature to obtain a speech feature; S6, inputting the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generating the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

[0008] Preferably, the lower facial encoder is used to extract and process local motion features, refine micro-expression details, and extract overall motion structure information to obtain global facial motion features; the upper facial encoder is used to process global facial motion features, focusing on overall motion trends; the speech content encoder and the speech emotion encoder are used to extract the semantic content and emotional time-frequency patterns of speech respectively; The convolution layer of the encoder is used to extract local spatial features of input data, capturing patterns within the neighborhood through a sliding window of the convolution kernel; the input data includes a facial motion sequence or a speech signal; The linear projection layer of the encoder is used to map the features extracted by convolution into a high-dimensional space and convert them into serialized feature vectors suitable for Transformer processing; The encoder's multi-layer Transformer layer is used to model long-range dependencies between features and capture global context information through the self-attention mechanism.

[0009] Preferably, the upper face decoder is used to generate global motion prediction; the lower face decoder is used to combine local features and speech features to generate refined motion prediction; The convolutional layer of the decoder is used to spatially reconstruct the encoded features and restore local structural information; The decoder's multi-layer Transformer layer is used to further optimize the temporal consistency of features and fuse encoder features or cross-modal information through the attention mechanism; The fully connected layers of the decoder are used to map high-dimensional features to specific output dimensions.

[0010] Preferably, the pre-trained speech content encoder and speech emotion encoder are two wav2vec2.0 models, which decouple the content and emotion of the speech and adopt a cross-reconstruction loss function for training and optimization.

[0011] Preferably, the emotional time-frequency enhancement feature is obtained by: First, the emotion feature is enhanced in the time domain by using a bidirectional long short-term memory network to obtain a forward and backward processing sequence of the bidirectional long short-term memory network BiLSTM, which is then fused with the emotion feature to obtain a time domain enhanced feature, which is expressed as: ; ; ; in, representing said emotional characteristics; 、 Represents BiLSTM forward processing , BiLSTM backward processing The obtained sequence; represents the time domain enhancement feature; Represents a join operation; Represents the dot product operation; Next, we enhance the time domain features To enhance the frequency domain, the expression is: ; ; in, Enhance the results for the channel; represents the activation function; represents a multilayer perceptron; represents the average pooling operation; Enhance results for space; The scale is Depth convolution operation; n represents the dimension; Finally, the channel enhancement result With the spatial enhancement results Fusion obtains the emotional time-frequency enhancement feature , the expression is: ; in, Represents a join operation; Represents a one-dimensional convolution operation.

[0012] Preferably, the speech feature is obtained by multi-scale fusion of the emotional time-frequency enhancement feature, the speech content feature and the style feature, and encoding the resultant feature through a speech encoder; wherein the speech encoder includes a one-dimensional convolutional layer, an emotional multi-head attention layer and a multi-layer Transformer layer; The one-dimensional convolution layer is used to perform a one-dimensional convolution operation on the multi-scale fused input features to capture the local changes of the speech signal on the time axis, and map the fused high-dimensional features to a number of channels suitable for subsequent attention mechanism processing through the convolution kernel sliding window; The emotional multi-head attention layer is used to dynamically capture the cross-modal correlation between emotional time-frequency enhancement features and speech content features through a multi-head attention mechanism, while combining style features to guide the attention weight to a specific style; and assign different attention weights to features at different time steps to highlight emotional key points and suppress irrelevant information; The multi-layer Transformer layer is used to capture the long-range dependencies of speech features over the entire sequence length through the self-attention mechanism, gradually extracting speech feature representations from local to global, and collaborating with the emotional multi-head attention layer to further integrate emotion, content, and style features to generate semantically consistent and emotionally rich speech representations.

[0013] Preferably, it also includes: The face reconstruction loss function is used to calculate the mean square error between the predicted face motion sequence and the real face motion sequence to improve the accuracy of generating 3D face animation; The temporal continuity of the generated 3D facial animation output is constrained by using a facial temporal smoothing loss function to avoid abrupt jitters and frame skipping in the generated 3D facial animation, thereby generating a smoother and more natural 3D facial animation. Among them, the face reconstruction loss function The expression is: ; in, represents the real face motion sequence of frame i; represents the face motion prediction sequence of the i-th frame; Represents the sequence dimension; The face temporal smoothing loss function The expression is: ; in, express expectations; Indicates the Frame face motion prediction sequence; represents the face motion prediction sequence of the i-th frame; Indicates the Frame real face motion sequence; represents the real face motion sequence of frame i; represents the L2 norm; The face reconstruction loss and the face temporal smoothing loss are weighted together to obtain the face motion loss function, which is expressed as: ; in, 、 Represents the weight coefficient.

[0014] The present invention also provides a three-dimensional facial animation generation device, comprising: A data acquisition unit, used for acquiring facial motion sequences and voice signals; a feature extraction unit, configured to capture global features and local features of facial motion respectively through a dual-layer facial encoder according to the facial motion sequence; the dual-layer facial encoder comprises an upper-layer facial encoder and a lower-layer facial encoder, each encoder comprising a convolutional layer, a linear projection layer, and a multi-layer Transformer layer; A facial motion feature unit, configured to generate a global facial motion prediction based on the global facial motion feature through an upper-layer facial decoder, and fuse the predicted global facial motion with the local facial motion feature to obtain a facial motion feature; A speech feature extraction unit is used to pass the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtain style features through one-hot encoding; A speech feature fusion unit is used to enhance the emotional feature based on a bidirectional long short-term memory network to obtain an emotional time-frequency enhancement feature, and perform multi-scale fusion and encoding with the speech content feature and the style feature to obtain a speech feature; A prediction generation unit is used to input the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generate the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

[0015] The present invention also provides a three-dimensional facial animation generation device, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the three-dimensional facial animation generation method as described above.

[0016] The present invention also provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the method for generating three-dimensional facial animation as described above is implemented.

[0017] In summary, compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a two-layer autoencoder to capture the global and local features of facial motion respectively, deeply explores the complex connection between facial motion and speech signals, and significantly improves the reliability and richness of generating three-dimensional facial animation.

[0018] The present invention designs a speech emotion time-frequency domain enhancement module, which performs detailed time-frequency domain enhancement processing on speech emotion features, fully explores and strengthens the emotional details in the speech signal, thereby maximizing the expressiveness of speech emotion features and providing diverse and reliable emotional information for the generation of three-dimensional facial animation.

[0019] The face motion loss function of the present invention further improves the naturalness and stability of generating three-dimensional face animation by jointly constraining the face reconstruction loss and the temporal smoothing loss.

[0020] This invention not only significantly improves the subtlety and naturalness of generated animations, but also effectively addresses the shortcomings of existing technologies in terms of emotional expression and diversity, providing a superior solution for the practical application of 3D virtual human technology. This invention is applicable to voice-driven 3D facial animation generation technology and can be widely used in scenarios such as intelligent customer service, virtual live broadcasts, and smart cities, providing a more natural and immersive experience for human-computer interaction.

[0021] The present invention outperforms the existing technologies in terms of indicators such as average vertex error, lip vertex error, average estimation error, coverage error and diversity index, and is effective in improving the accuracy and diversity of 3D facial animation generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A flowchart of a method for generating three-dimensional facial animation provided in Example 1.

[0024] Figure 2 This is a structural diagram of a three-dimensional facial animation generation method provided in Example 1.

[0025] Figure 3 This is a schematic diagram of a three-dimensional facial animation generation device provided in Example 2.

[0026] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Example 1 Embodiment 1 of the present invention provides a 3D facial animation generation method, which can be implemented by a 3D facial animation generation device (hereinafter referred to as a generation device), and in particular, executed by one or more processors in the generation device.

[0029] In this embodiment, the generating device may be an electronic device equipped with a processor, the processor having a computer program of the three-dimensional facial animation generating method and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0030] like Figure 1As shown, a three-dimensional facial animation generation method includes steps S1 to S6.

[0031] S1, obtain facial motion sequence and speech signal.

[0032] In this step, synchronously collected facial motion sequences (such as three-dimensional key point coordinates, mesh vertex displacements) and speech signals (original audio waveforms or spectra) are obtained.

[0033] S2, according to the facial motion sequence, captures the local features and global features of facial motion respectively through a dual-layer facial encoder; the dual-layer facial encoder includes an upper-layer facial encoder and a lower-layer facial encoder, each encoder includes a convolution layer, a linear projection layer and a multi-layer Transformer layer.

[0034] like Figure 2 As shown, the lower-layer facial encoder is used to extract and process local motion features, refine micro-expression details, and extract overall motion structure information to obtain global facial motion features; the upper-layer facial encoder is used to process global facial motion features and focus on overall motion trends.

[0035] In this embodiment, each encoder includes a convolutional layer, a linear projection layer, and a multi-layer Transformer layer.

[0036] Among them, the convolution layer of the encoder is used to extract local spatial features of the input data (facial motion sequence or speech signal) (such as pixel-level changes in the corners of the mouth), and capture patterns in the neighborhood (such as edges, textures, etc.) through the convolution kernel sliding window.

[0037] The linear projection layer of the encoder is used to map the features extracted by convolution to a high-dimensional space and convert them into serialized feature vectors suitable for Transformer processing; this is usually achieved through 1×1 convolution or fully connected layers, adjusting the feature dimensions to match the input requirements of the subsequent Transformer.

[0038] The encoder's multi-layer Transformer layer is used to model long-range dependencies between features and capture global context information through the self-attention mechanism.

[0039] Assume that the acquired face motion sequence is ; represents the real face motion sequence of frame i, Represents the sequence dimension.

[0040] Lower-layer facial encoder: First, the lower-layer facial encoder extracts local features to capture fine-grained facial expression changes and obtain local facial motion features, such as a raised corner of the mouth or a trembling eyebrow. The operation expression is as follows: ； in, Represents the local motion features of the face, represents the lower layer facial encoder, the input dimension is , the output dimension is set to 256, It can be composed of a one-dimensional convolution layer, a linear projection layer, and 8 Transformer layers.

[0041] Upper facial encoder: The upper facial encoder abstracts the local motion features of the face to obtain the global motion features of the face, and extracts the overall motion structure information, such as global actions such as smiling and frowning. This operation is shown in the following formula: ； in, Represents the global motion characteristics of the face, Represents the upper face encoder, set It can be composed of a one-dimensional convolution layer, a linear projection layer, and an 8-layer Transformer layer. The input facial local motion features The dimension of is 256, and the output dimension is 128.

[0042] S3, generating a global face motion prediction through an upper-layer facial decoder according to the global face motion feature, and fusing it with the local face motion feature to obtain a face motion feature.

[0043] In this embodiment, the upper-layer facial decoder is used to generate global motion prediction; the lower-layer facial decoder is used to combine local features and speech features to generate refined motion prediction.

[0044] The convolutional layer of the decoder is used to spatially reconstruct the encoded features and restore local structural information; The decoder's multi-layer Transformer layer is used to further optimize the temporal consistency of features and fuse encoder features or cross-modal information through the attention mechanism; The fully connected layers of the decoder are used to map high-dimensional features to specific output dimensions.

[0045] The S3 is specifically: S31, for global motion features of the face Using the upper layer facial decoder, the global motion prediction of the face is generated. The process can be expressed as: ; in, represents the global motion prediction of the face, Denotes the upper face decoder. Assume that the upper face decoder consists of a one-dimensional convolutional layer, an 8-layer Transformer layer, and a fully connected layer. The input global motion features of the face If the dimension is 128, the output dimension is 256.

[0046] S32, then predict the global motion of the face and local motion features of the face Perform connection operations to obtain facial motion features , the operation can be expressed as: ; in, Represents the facial motion features, represents the global motion prediction of the face, Represents the local motion features of the face, Represents a join operation.

[0047] S4, passing the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtaining style features through one-hot encoding.

[0048] In this embodiment, the pre-trained speech content encoder and speech emotion encoder are two wav2vec2.0 models, which decouple the speech content and emotion through a speech content emotion decoupling coding method similar to Emotalk, and adopt a cross-reconstruction loss function for training and optimization.

[0049] In this embodiment, the speech content-emotion decoupling coding method aims to simultaneously extract content and emotion features from speech signals while ensuring their independence. This method typically employs a dual-encoder architecture, with one encoder responsible for extracting content features and the other for extracting emotion features. By designing a specific loss function (such as the cross-reconstruction loss) and training strategy, the two encoders can focus on different aspects of speech, thereby achieving decoupling of content and emotion.

[0050] To ensure the independence of content and emotion features, a cross-reconstruction loss is introduced. Specifically, the cross-reconstruction loss uses the features extracted by the content encoder and the output of the emotion encoder (or a randomly generated emotion label) to reconstruct the speech signal and calculates a reconstruction error. Simultaneously, it also uses the features extracted by the emotion encoder and the output of the content encoder (or a randomly generated content label) to reconstruct the speech signal and calculate another reconstruction error. By minimizing these two reconstruction errors, the model is forced to learn independent content and emotion features.

[0051] The speech content encoder is used to extract the semantic content of speech (such as pronunciation, speaking rate, intonation, etc.), ignoring emotional information and focusing on "what is said". The output features are used to drive the lip synchronization and basic movements (such as lip shape and tongue movement) of facial animation. This process can be expressed as: ; in, is the speech content feature; is a speech content encoder; For the voice signal.

[0052] During the pre-training phase, the model is trained via a cross-reconstruction loss function to make it insensitive to emotion changes (e.g., mapping the same speech content with different emotions to the same feature space).

[0053] By decoupling training, the speech content encoder can be made robust to emotion (e.g., mapping “happy” and “sad” versions of “hello” to the same content features).

[0054] The speech emotion encoder is used to extract emotional features (such as anger, joy, sadness, etc.) from speech signals, ignoring the specific semantic content and focusing on "how to say it". The output features are used to enhance the emotional expressions of facial animation (such as raised eyebrows and drooping mouth corners). The process can be expressed as: ; in, For emotional characteristics; It is a speech emotion encoder; For the voice signal.

[0055] During the pre-training stage, the model focuses only on emotional information through emotional supervision signals (such as annotated emotional labels) and content adversarial loss.

[0056] Through emotional supervision signals and content adversarial training, the emotion encoder is ensured to be insensitive to content (e.g., “angry” speech with different contents is mapped to similar emotional features).

[0057] A two-stage training strategy can be adopted. In the first stage, the model is pre-trained using a large amount of unlabeled speech data to learn general speech representations. In the second stage, the model is fine-tuned using a small amount of labeled speech data to adapt to a specific task (such as emotion classification or speech synthesis).

[0058] In addition, voice style (such as speaker ID, dialect, and speaking speed) is represented through one-hot encoding or embedding vectors to control the personalized performance of animations (such as the pronunciation habits of specific characters). For example, the input is a style label (such as "cartoon style" or "realistic style"); the output is a style feature vector, which is mapped to a continuous space through the embedding layer.

[0059] S5, based on the bidirectional long short-term memory network, the emotional feature is enhanced to obtain the emotional time-frequency enhancement feature, and the emotional time-frequency enhancement feature is multi-scale fused and encoded with the speech content feature and the style feature to obtain the speech feature.

[0060] In this step, in order to enhance the delicacy and diversity of facial and voice emotional expression, the present invention uses emotional features Perform time-frequency domain feature enhancement.

[0061] Specifically, first, the emotion feature is enhanced in the time domain dimension through the bidirectional long short-term memory network to obtain the forward and backward processing sequence of the bidirectional long short-term memory network BiLSTM, and then it is fused with the emotion feature to obtain the time domain enhanced feature, which is expressed as: ; ; ; in, representing said emotional characteristics; 、 Represents BiLSTM forward processing , BiLSTM backward processing The obtained sequence; represents the time domain enhancement feature; Represents a join operation; Represents the dot product operation; Next, we enhance the time domain features To enhance the frequency domain, the expression is: ; ; in, Enhance the results for the channel; represents the activation function; represents a multilayer perceptron; represents the average pooling operation; Enhance results for space; The scale is Depth convolution operation; n represents the dimension; Finally, the channel enhancement result With the spatial enhancement results Fusion obtains the emotional time-frequency enhancement feature , the expression is: ; in, Represents a join operation; Represents a one-dimensional convolution operation.

[0062] At this point, the voice content features have been obtained , emotional time-frequency enhancement features For speech style information, we use single-hot encoding to get style features Next, these three features are multi-scale fused to obtain the speech fusion feature , the operation process is as follows: ； in, represents the dot product operation, Represents a join operation.

[0063] Next, use the speech encoder to integrate the speech features The speech features are encoded, and the speech encoder includes a one-dimensional convolution layer, an emotional multi-head attention layer, and a multi-layer Transformer layer (e.g., 6 layers).

[0064] The operation is as follows: ; in, is the voice feature; For the speech encoder.

[0065] The one-dimensional convolution layer of the speech encoder is used to perform a one-dimensional convolution operation on the multi-scale fused input features to capture the local changes of the speech signal on the time axis. The convolution kernel sliding window is used to map the fused high-dimensional features to a number of channels suitable for subsequent attention mechanism processing. The emotion multi-head attention layer of the speech encoder is used to dynamically capture the cross-modal correlation between the emotion time-frequency enhancement features and the speech content features through a multi-head attention mechanism. At the same time, it combines style features to guide the attention weight to a specific style. It also assigns different attention weights to features at different time steps to highlight emotional key points and suppress irrelevant information. The multi-layer Transformer layer of the speech encoder is used to capture the long-range dependencies of speech features over the entire sequence length through the self-attention mechanism, gradually extracting speech feature representations from local to global, and collaborating with the emotional multi-head attention layer to further integrate emotion, content, and style features to generate semantically consistent and emotionally rich speech representations.

[0066] S6, inputting the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generating the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

[0067] After the above operations, we have obtained the facial motion features and speech features. After connecting the two, we use the lower-level facial decoder to obtain facial motion prediction. The operation is shown in the following formula: ; in, represents face motion prediction, Represents the voice features, Represents the facial motion features, Indicates a connection operation. Represents the lower-level facial decoder. Assume that the lower-level facial decoder consists of a one-dimensional convolutional layer, 8-layer Transformer layer, and a fully connected layer. The input dimension is 256, then the output dimension is .

[0068] In another preferred embodiment, in order to improve the accuracy of generating 3D facial animation, the method further comprises: calculating the mean square error between the predicted facial motion sequence and the actual facial motion sequence using a facial reconstruction loss function; The temporal continuity of the generated 3D facial animation output is constrained by using a facial temporal smoothing loss function to avoid abrupt jitters and frame skipping in the generated 3D facial animation, thereby generating a smoother and more natural 3D facial animation. Among them, the face reconstruction loss function The expression is: ; in, represents the real face motion sequence of frame i; represents the face motion prediction sequence of the i-th frame; Represents the sequence dimension; The face temporal smoothing loss function The expression is: ; in, express expectations; Indicates the Frame face motion prediction sequence; represents the face motion prediction sequence of the i-th frame; Indicates the Frame real face motion sequence; represents the real face motion sequence of frame i; represents the L2 norm; The face reconstruction loss and the face temporal smoothing loss are weighted together to obtain the face motion loss function, which is expressed as: ; in, 、 Represents the weight coefficient, if , .

[0069] In another preferred embodiment, in order to further verify the effect of the present invention, the 3DMEAD dataset is used to evaluate the model performance from the accuracy and diversity of generated facial animations.

[0070] This dataset contains 3D reconstructions of 47 English-speaking subjects, covering eight emotion types and three intensity levels. The emotion categories include neutral, happy, sad, surprised, fearful, disgusted, angry, and contemptuous. Except for the neutral category, each emotion category has three intensity levels: weak, medium, and strong. Each subject provided 30 short sentences for each of the seven basic emotions, 10 for each intensity level, and 40 sentences for neutral emotion.

[0071] The training is divided into two stages: In the first stage, the uncertainty-guided dual-layer facial autoencoder is trained on the 3DMEAD dataset. Among them, 32 characters are used for training, 7 characters are used for validation, and 8 characters are used for testing. The critical parameters are The value is 0.3, the weight parameter in the first stage loss function In the second stage, the speech time-frequency emotion feature enhancement encoder is trained on the same dataset, and combined with the uncertainty-guided dual-layer facial autoencoder trained in the first stage to generate facial motion prediction. Among them, the weight parameter of the loss function in the second stage is , The first stage aims to learn facial motion priors through an uncertainty-guided dual-layer facial autoencoder. The second stage enhances the encoder with speech time-frequency emotion features to deeply explore the potential connection between speech emotion features and facial animation.

[0072] In the experiment, the most advanced methods were selected to conduct a comparative experiment on detection effects, specifically: Existing method 1: Based on the Transformer architecture, high-quality lip synchronization is achieved. For details, see: Fan Y, Lin Z, Saito J, et al. Faceformer: Speech-driven 3d facial animation with transformers[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition. Los Alamitos: IEEE Computer SocietyPress, 2022: 18749-18758.

[0073] Existing method 2: Using the self-supervised speech representation model HuBERT to achieve 3D facial animation generation. See: Haque KI, Yumak Z. Facexhubert: Text-less speech-driven e (x)pressive 3d facial animation synthesis using self-supervised speechrepresentation learning-ing[C] / / Proceedings of the International Conference onMultimodal Interaction. New York: Association for Computing Machinery, 2023:282-291.

[0074] Existing method 3: Convert facial animation generation into a discrete space vector query task to achieve voice-driven facial animation generation. See: Xing J, Xia M, Zhang Y, et al. Codetalker: Speech-driven 3d facial animation with discrete motion prior[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. Los Alamitos: IEEEComputer Society Press, 2023: 12780-12790.

[0075] Existing method 4: Generate speech-lip synchronization facial animation based on diffusion model to improve the naturalness and accuracy of expression. Stan S, Haque KI, Yumak Z, et al. Facediffuser: Speech-driven 3dfacial animation synthesis using diffusion[C] / / Proceedings of the ACMSIGGRAPH Conference on Motion, Interaction and Games. New York: Association for Computing Machinery, 2023: 1-11. Existing method 5: Using uncertainty modeling and multimodal fusion to achieve lip synchronization and facial emotion expression. See: Wu S, Haque KI, Yumak Z, et al. ProbTalk3D: Non-DeterministicEmotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE[C] / / Proceedings of the ACM SIGGRAPH Conference on Motion, Interaction, andGames. New York: Association for Computing Machinery, 2024: 1-12. This experiment uses the average vertex error and lip vertex error to reflect the accuracy of generating 3D facial animation, and uses the average estimation error, coverage error and diversity index to reflect the diversity of generating 3D facial animation.

[0076] Average vertex error: Calculates the average Euclidean distance between the face motion prediction frames and the real face motion data in the test set.

[0077] Lip vertex error: First calculate the Euclidean distance between the vertices of the mouth area of each predicted frame, and then calculate the average value among all generated frames.

[0078] Average estimation error: Calculate the average value of the facial motion prediction sampling distribution to determine how close it is to the actual facial motion data.

[0079] Coverage error: Calculate the LVE between 10 face prediction samples and the real data and take the minimum value, and then perform the average operation.

[0080] Diversity Index: Given the same speech input, the generated facial animation samples are randomly divided into two subsets. The average Euclidean distance between the corresponding samples is calculated, and the average value of all speech inputs is used to obtain the diversity index, which is used to measure the diversity of facial animation.

[0081] The same experiment was conducted three times. The average vertex error, lip vertex error, average estimation error, coverage error and diversity index of the 3D facial animation generated by the experimental test set are shown in Table 1.

[0082] Table 1 Experimental comparison results of the method of the present invention and five existing methods

[0083] The experimental results show that when this method is used, all indicators are better than the results obtained by existing methods 1-5.

[0084] In terms of facial accuracy, the proposed method surpasses the best existing methods by 2.51% in average vertex error and 2.46% in lip vertex error, respectively. This is primarily due to the use of a speech emotion time-frequency domain enhancement module. This module, while taking into account the multi-scale characteristics of speech, can deeply explore the multi-dimensional connection between speech emotion features and 3D facial animation, resulting in more detailed and natural 3D facial animation.

[0085] In terms of generating facial diversity indicators, the method of the present invention outperforms other methods in terms of average estimation error, coverage error, and diversity index. This shows that the method proposed in this scheme has significantly improved the richness and delicacy of dynamic changes in the face, and can generate more accurate and diverse three-dimensional facial animations. The main reason is that this scheme adopts an uncertainty quantization mechanism for the dimensionality reduction representation of facial motion features. According to the level of uncertainty indicators, different quantization methods are dynamically adopted. Under the premise of ensuring the accuracy of generated facial animations, this mechanism provides greater flexibility and expressiveness for the quantification of facial motion features, thereby realizing the generation of three-dimensional facial animations with emotional diversity.

[0086] Judging from the experimental results, the voice-driven three-dimensional facial animation generation method designed by the present invention has certain theoretical significance and practical application value, and the experiment verifies the effectiveness of the method proposed by the present invention.

[0087] The present invention adopts a two-layer encoder division of labor mechanism: upper layer (global) → lower layer (local), and transfers global motion constraints to local details through feature fusion (such as residual connection), preventing local features from deviating from the overall rationality.

[0088] This invention uses speech feature enhancement. The emotional time-frequency features are enhanced with LSTM to enhance the temporal dynamics, and then integrated with the content / style features to ensure that the speech-driven expressions are natural and emotional.

[0089] The present invention adopts a decoder hierarchical linkage mechanism: the lower-level decoder integrates facial motion features and voice features to achieve multi-modal driven refined animation generation.

[0090] In practical applications, the present invention can be used in scenarios such as intelligent customer service, virtual live broadcasting, and smart cities. Taking intelligent customer service as an example, when a user interacts with virtual customer service through voice, the system first extracts the content and emotional information from the voice signal through the voice content encoder and the voice emotion encoder, and enhances the expressiveness of emotional features through the voice emotion time-frequency domain enhancement operation. Subsequently, the system combines the facial motion features extracted by the dual-layer facial encoder to generate a three-dimensional facial animation that is highly matched with the voice content and emotion. In this way, virtual customer service can interact with users in a more natural and emotional way, thereby improving the user experience.

[0091] In summary, compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a two-layer autoencoder to capture the global and local features of facial motion respectively, deeply explores the complex connection between facial motion and speech signals, and significantly improves the reliability and richness of generating three-dimensional facial animation.

[0092] The present invention designs a speech emotion time-frequency domain enhancement module, which performs detailed time-frequency domain enhancement processing on speech emotion features, fully explores and strengthens the emotional details in the speech signal, thereby maximizing the expressiveness of speech emotion features and providing diverse and reliable emotional information for the generation of three-dimensional facial animation.

[0093] This invention achieves a technological breakthrough in speech-driven 3D facial animation generation through the collaborative work of modules such as a dual-layer facial encoder, a speech emotion time-frequency domain enhancement module, and a facial motion loss function. This not only addresses the shortcomings of existing technologies in terms of emotional expression and diversity, but also significantly improves the naturalness and stability of the generated animation through multi-dimensional feature enhancement and joint loss constraints. This invention can be widely applied in various human-computer interaction scenarios, providing users with a more immersive experience.

[0094] Example 2 like Figure 3 As shown, the second embodiment of the present invention further provides a three-dimensional facial animation generation device, comprising: A data acquisition unit, used for acquiring facial motion sequences and voice signals; a feature extraction unit, configured to capture global features and local features of facial motion respectively through a dual-layer facial encoder according to the facial motion sequence; the dual-layer facial encoder comprises an upper-layer facial encoder and a lower-layer facial encoder, each encoder comprising a convolutional layer, a linear projection layer, and a multi-layer Transformer layer; A facial motion feature unit, configured to generate a global facial motion prediction based on the global facial motion feature through an upper-layer facial decoder, and fuse the predicted global facial motion with the local facial motion feature to obtain a facial motion feature; A speech feature extraction unit is used to pass the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtain style features through one-hot encoding; A speech feature fusion unit is used to enhance the emotional feature based on a bidirectional long short-term memory network to obtain an emotional time-frequency enhancement feature, and perform multi-scale fusion and encoding with the speech content feature and the style feature to obtain a speech feature; A prediction generation unit is used to input the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generate the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

[0095] Example 3 The third embodiment of the present invention further provides a three-dimensional facial animation generation device, which includes a memory and a processor. The memory stores a computer program, and the computer program can be executed by the processor to implement the three-dimensional facial animation generation method as described above.

[0096] Example 4 The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the three-dimensional facial animation generation method as described above is implemented.

[0097] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0098] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0099] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0100] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0101] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0102] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0103] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0104] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for generating a three-dimensional facial animation, characterized in that: include: S1, obtains facial motion sequence and speech signal; S2, according to the facial motion sequence, capturing global facial motion features and local facial motion features respectively through a dual-layer facial encoder; the dual-layer facial encoder includes an upper-layer facial encoder and a lower-layer facial encoder, each encoder including a convolutional layer, a linear projection layer, and a multi-layer Transformer layer; S3, generating a global face motion prediction through an upper-layer facial decoder based on the global face motion feature, and fusing it with the local face motion feature to obtain a face motion feature; S4, passing the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtaining style features through one-hot encoding; S5, performing feature enhancement on the emotional feature based on a bidirectional long short-term memory network to obtain an emotional time-frequency enhancement feature, and performing multi-scale fusion and encoding with the speech content feature and the style feature to obtain a speech feature; S6, inputting the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generating the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

2. A 3D facial animation generation method according to claim 1, characterized in that ,The lower layer facial encoder is used to extract and process local ,motion features, refine micro-expression details, and extract the overall ,motion structure information to obtain the global motion features of the ,face; The upper facial encoder is used to process the global features of facial motion and focus on the overall motion trend; the speech content encoder and the speech emotion encoder are used to extract the semantic content and emotional time-frequency pattern of speech respectively; The convolution layer of the encoder is used to extract local spatial features of input data, capturing patterns within the neighborhood through a sliding window of the convolution kernel; the input data includes a facial motion sequence or a speech signal; The linear projection layer of the encoder is used to map the features extracted by convolution into a high-dimensional space and convert them into serialized feature vectors suitable for Transformer processing; The encoder's multi-layer Transformer layer is used to model long-range dependencies between features and capture global context information through the self-attention mechanism.

3. A 3D facial animation generation method according to claim 1, characterized in that The upper face decoder is used to generate global motion prediction; the lower face decoder is used to combine local features and speech features to generate refined motion prediction; The convolutional layer of the decoder is used to spatially reconstruct the encoded features and restore local structural information; The decoder's multi-layer Transformer layer is used to further optimize the temporal consistency of features and fuse encoder features or cross-modal information through the attention mechanism; The fully connected layers of the decoder are used to map high-dimensional features to specific output dimensions.

4. A method for generating three-dimensional facial animation according to claim 1, characterized in that The pre-trained speech content encoder and speech emotion encoder are two wav2vec2.0 models, which decouple the content and emotion of speech and adopt cross-reconstruction loss function for training and optimization.

5. A 3D facial animation generation method according to claim 1, characterized in that ,The emotional time-frequency enhancement features are obtained by the following ,methods: First, the emotion feature is enhanced in the time domain by using a bidirectional long short-term memory network to obtain a forward and backward processing sequence of the bidirectional long short-term memory network BiLSTM, which is then fused with the emotion feature to obtain a time domain enhanced feature, which is expressed as: ； ； ； in, representing said emotional characteristics; 、 Represents BiLSTM forward processing , BiLSTM backward processing The obtained sequence; represents the time domain enhancement feature; Represents a join operation; Represents the dot product operation; Next, we enhance the time domain features To enhance the frequency domain, the expression is: ； ； in, Enhance the results for the channel; represents the activation function; represents a multilayer perceptron; represents the average pooling operation; Enhance results for space; The scale is Depth convolution operation; n represents the dimension; Finally, the channel enhancement result With the spatial enhancement results Fusion obtains the emotional time-frequency enhancement feature , the expression is: ； in, Represents a join operation; Represents a one-dimensional convolution operation.

6. A 3D facial animation generation method according to claim 1, characterized in that The speech feature is obtained by multi-scale fusion of the emotional time-frequency enhancement feature, the speech content feature and the style feature, and encoding them through a speech encoder; wherein the speech encoder includes a one-dimensional convolution layer, an emotional multi-head attention layer and a multi-layer Transformer layer; The one-dimensional convolution layer is used to perform a one-dimensional convolution operation on the multi-scale fused input features to capture the local changes of the speech signal on the time axis, and map the fused high-dimensional features to a number of channels suitable for subsequent attention mechanism processing through the convolution kernel sliding window; The emotional multi-head attention layer is used to dynamically capture the cross-modal correlation between emotional time-frequency enhancement features and speech content features through a multi-head attention mechanism, while combining style features to guide the attention weight to a specific style; and assign different attention weights to features at different time steps to highlight emotional key points and suppress irrelevant information; The multi-layer Transformer layer is used to capture the long-range dependencies of speech features over the entire sequence length through the self-attention mechanism, gradually extracting speech feature representations from local to global, and collaborating with the emotional multi-head attention layer to further integrate emotion, content, and style features to generate semantically consistent and emotionally rich speech representations.

7. A method for generating three-dimensional facial animation according to claim 1, characterized in that , also includes: The face reconstruction loss function is used to calculate the mean square error between the predicted face motion sequence and the real face motion sequence to improve the accuracy of generating 3D face animation; The temporal continuity of the generated 3D facial animation output is constrained by using a facial temporal smoothing loss function to avoid abrupt jitters and frame skipping in the generated 3D facial animation, thereby generating a smoother and more natural 3D facial animation. Among them, the face reconstruction loss function The expression is: ； in, represents the real face motion sequence of frame i; Represents the face motion prediction sequence of the i-th frame; Represents the sequence dimension; The face temporal smoothing loss function The expression is: ； in, express expectations; Indicates the Frame face motion prediction sequence; Represents the face motion prediction sequence of the i-th frame; Indicates the Frame real face motion sequence; represents the real face motion sequence of frame i; represents the L2 norm; The face reconstruction loss and the face temporal smoothing loss are weighted together to obtain the face motion loss function, which is expressed as: ； in, 、 Represents the weight coefficient.

8. A 3D facial animation generating device, characterized in that: include: A data acquisition unit, used for acquiring facial motion sequences and voice signals; a feature extraction unit, configured to capture global features and local features of facial motion respectively through a dual-layer facial encoder according to the facial motion sequence; the dual-layer facial encoder comprises an upper-layer facial encoder and a lower-layer facial encoder, each encoder comprising a convolutional layer, a linear projection layer, and a multi-layer Transformer layer; A facial motion feature unit, configured to generate a global facial motion prediction based on the global facial motion feature through an upper-layer facial decoder, and fuse the predicted global facial motion with the local facial motion feature to obtain a facial motion feature; A speech feature extraction unit is used to pass the speech signal through two pre-trained speech content encoders and speech emotion encoders to obtain speech content features and emotion features respectively, and obtain style features through one-hot encoding; A speech feature fusion unit is used to enhance the emotional feature based on a bidirectional long short-term memory network to obtain an emotional time-frequency enhancement feature, and perform multi-scale fusion and encoding with the speech content feature and the style feature to obtain a speech feature; A prediction generation unit is used to input the facial motion features and the speech features into the lower-level facial decoder to obtain facial motion prediction, and then generate the final three-dimensional facial animation; wherein each decoder includes a convolutional layer, a multi-layer Transformer layer and a fully connected layer.

9. A 3D facial animation generating device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a three-dimensional facial animation generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a three-dimensional facial animation generation method according to any one of claims 1 to 7 is implemented.

Images