Voice-driven facial animation simulation method based on facial muscle linkage

Abstract

The invention discloses a voice-driven facial animation simulation method based on facial muscle linkage, and belongs to the field of artificial intelligence, and the method comprises the steps: S1, constructing a PPMF encoder; s2, constructing a decoder based on FDCP to decode FP features provided by the PPMF so as to obtain a facial animation; s3, training a 3D face animation framework DCPTalk driven by voice; s4, optimizing the model; and S5, quantitatively evaluating the model. According to the invention, a DCPTalk framework is provided, and the Mouth2Face is provided based on the linkage characteristic of the facial muscle group. Motion of the mouth has strong correlation with voice signals, and can be synthesized with sound track dynamics easily. In order to further enhance the details of the facial movement, the Refine Decoder is adopted to simulate the skin deformation of the surface layer to refine the facial animation. Inherent body characteristics and body characteristics related to facial muscle group movement are embedded into Mouth2Face, a personalized facial muscle control system is constructed, and meanwhile, an external driving signal is modulated by using a speaking style. Qualitative and quantitative experiments and user research show that the DCPTalk is superior to the most advanced method in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a voice-driven facial animation simulation method based on facial muscle linkage. Background Art

[0002] Speech-driven 3D facial animation has become a focus of important scientific interest in both academic and industrial fields. Its applications include film production, computer games, education and training, and telemedicine. Speech-driven 3D facial animation techniques, including 3DMM-based methods and mesh-based methods, are highly diverse. 3DMM-based methods usually predict intermediate 3DMM coefficients under speech conditions and transform them into 3D space. However, these 3DMM coefficients, as intermediate variables, lack clear semantics and may cause misalignment between lip movements and driving audio.

[0003] Facial expressions are produced by traction forces exerted by 268 muscles on soft tissues. The muscles in the mouth area are closely connected to the surrounding muscles and are the basic drivers of facial expression changes, ensuring coordinated changes between the movements of different facial areas. In addition, the description of expressive details is closely related to the deformation of the surface layer of the skin. However, the synthesis of facial expressions is not just a process of combining facial muscle activity with surface skin deformation; it is also influenced by individual factors (factors), such as inherent physical characteristics and acquired speaking habits and speaking styles. For people with elongated or wider facial shapes, the traction of muscles may be reduced because longer muscle fibers have weaker traction than shorter muscle fibers. The increase of collagen under the facial skin further limits the stretchability of these fibers.

[0004] Existing methods usually simplify the facial animation generation task to infinitely thin surface skin deformation without underlying structure, thus ignoring the intricate and personalized dynamics of facial muscle activity. The structure and activity of facial muscles have been widely studied and applied in various fields. However, the interactions and forces between these muscles have not been thoroughly quantified. Summary of the invention

[0005] In order to solve the problem that the above-mentioned prior art ignores the intricacies and personalized dynamics of facial muscle activities and simplifies the facial animation generation task into infinitely thin surface skin deformation without the problem of underlying structure, the present invention provides a voice-driven facial animation simulation method based on facial muscle linkage.

[0006] To implement the above technical solution, the present invention provides a voice-driven facial animation simulation method based on facial muscle linkage, comprising the steps of:

[0007] S1: Construct a PPMF encoder, which consists of an audio feature extractor and a pseudo facial key point extractor, and uses a module similar to the Transformer decoder to fuse and align personalized pseudo facial key point features F L and audio feature F A ;

[0008] S2: Construct a decoder based on FDCP to decode the F P Features are used to obtain facial animation. The decoder consists of Mouth Mapping, Mouth2Face, and Refine Decoder. The three stages of synthesizing mouth movements from driving signals, using mouth movements to evoke facial animation, and refining facial animation are implemented by Mouth Mapping, Mouth2Face, and Refine Decoder respectively.

[0009] S3: Training the voice-driven 3D face animation framework DCPTalk, first using the loss function Train the Mouth2Face module to establish the mapping rules between mouth movement and facial animation; then fix the trained Mouth

[0010] 2Face parameters, and began to train other components of DCPTalk, and gave the loss functions for training Mouth2Face and other components respectively; the loss function includes reconstruction loss Speed ​​loss and facial keypoint loss

[0011] S4: Model optimization, introducing BIWI, Multiface and VOCASET to provide comprehensive analysis and optimization for DCPTalk, and comparing VOCA, MeshTalk, FaceFormer, CodeTalker, FaceDiffuse, DiffSpeaker, TalkingStyle and SelfTalk with the method of the present invention; then training DCPTalk on a single NVIDIA A100 GPU;

[0012] S5: Quantitative evaluation of the model, based on the methods of FaceFormer, CodeTalker and SelfTalk, by calculating the lip vertex error (LVE) to evaluate the synchronization between speech content and lip movement.

[0013] Further, in step S1, the pseudo facial key point feature F L and audio feature F A The fusion steps include:

[0014] S1a: Using a multi-head self-attention layer with linear bias (ALiBi) for F L to process;

[0015] S1b: Use a multi-head cross attention layer to convert the F obtained by the self-attention layer A and With F L Alignment;

[0016] S1c: After the feed-forward layer, the personalized pseudo multimodal feature F is obtained P =[f1 P ,…,f t P ,…,f T P ]∈R T×d .

[0017] Furthermore, the audio feature extractor is composed of an audio encoder A and an audio feature projection layer F A The composition is expressed as follows:

[0018] A=Audio Encoder(χ;θ A )

[0019] F A =Audio Projection(Α;ψ A ).

[0020] Furthermore, the pseudo facial key point extractor includes an audio encoder and an Audio2lmk decoder, and finally outputs 3D pseudo facial key points L = [l1,…,l t ,…,l T ]∈R T×68×3 ; Use personalized factors P∈R N Personalized facial key point modulation: Using the personalized factor P, the pseudo facial key point L in the representation space is modulated by element-level additive modulation. The formula is as follows:

[0021] L = Audio2lmk Decoder(Α;θ L )

[0022] F L =Personlized Modulation(L,P;ψ L ).

[0023] Furthermore, in step S2, the process of Mouth Mapping for synthesizing accurate mouth movements is expressed by the formula:

[0024]

[0025] Mouth2Face uses the mouth encoder to convert mouth movements Converted to a signal Q that controls facial muscle activity by using an element-wise quantization function Discrete facial muscle control signal Q to obtain facial muscle control command Activate relevant facial muscles to achieve facial animation The process is expressed by the formula:

[0026]

[0027]

[0028]

[0029] Refine Decoder is based on F P The refinement stage predicts the displacement of each vertex and then adds the vertex displacement to the vertex position of the face caused by the mouth movement in a basic way by formulating:

[0030]

[0031] Furthermore, Mouth2Face uses mouth movements and the facial animation Y=[y1,…,y t ,…,y T ]∈R T×V×3 Conduct training.

[0032] Further, in step S3, the process of DCPTalk training through the loss function includes Mouth2Face module training, which is expressed by the formula:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] Further, in step S4, for the pre-trained models lacking BIWI, VOCASET or Multiface, their official source codes are used to retrain their models.

[0040] Furthermore, in step S4, the model parameters are iteratively updated using the Adam optimizer with beta1 = 0.9, beta2 = 0.999 and a learning rate of 1 × 10 -4 ; The feature dimension d is 256; The facial muscle control commands H and P are set to 32 and 128 respectively; The loss function, λ0=1.0, λ lmk =1×10 -5 , λ1=0.1, λ2=2.0, λ3=1.0, β1=0.3, β2=4.0 and β3=10.0.

[0041] In summary, the present invention has the following beneficial effects compared with the prior art:

[0042] The present invention proposes a new framework called DCPTalk to simulate the complex dynamics of facial muscle activities and depict personalized facial animations. Based on the linkage relationship of facial muscles, the present invention proposes Mouth2Face to simulate the facial muscle control system and generate realistic and coordinated facial animations induced by mouth movements. Mouth movements are highly correlated with speech signals and can be easily synthesized with vocal tract dynamics. In order to further enhance the details of facial movements, the present invention uses surface skin deformation to refine the facial animations generated by Mouth2Face, and uses Refine Decoder to simulate the surface skin deformation to refine the facial animations. In addition, personalized factors (factors), including intrinsic body characteristics and acquired speaking style, directly determine the uniqueness and realism of facial animations. The inherent body characteristics and body characteristics related to the movement of facial muscle groups are embedded in Mouth2Face to construct a personalized facial muscle control system, while the speaking style is used to modulate the external driving signal. Qualitative and quantitative experiments and user studies show that DCPTalk outperforms existing state-of-the-art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 A flow chart of a voice-driven facial animation simulation method based on facial muscle linkage provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the DCPTalk framework. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form can also include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] See also Figure 1 As shown, the present invention provides a voice-driven facial animation simulation method based on facial muscle linkage, wherein DCPTalk is a voice-driven 3D facial animation personalized facial dynamic coupling characteristic, comprising the steps of:

[0049] S1: Construct a PPMF encoder, which consists of an audio feature extractor and a pseudo facial key point extractor;

[0050] As a preferred embodiment, the audio feature extractor is composed of an audio encoder A and an audio feature projection layer F A Specifically, the audio encoder is an improved large-scale pre-trained self-supervised speech feature extraction model WavLM, which contains a temporal convolution module (TCN) encoder and a Transformer encoder. A linear interpolation layer is embedded after the CNN encoder of WavLM to adapt to the frame rate of the target facial animation. During training, the CNN encoder parameters are fixed and the Transform encoder parameters are optimized. The linear mapping layer is responsible for converting the high-dimensional features extracted by WavLM into lower-dimensional features and adapting them to the facial animation generation task. The formula of this process is expressed as follows:

[0051] A=Audio Encoder(χ;θ A ) (1)

[0052] F A =Audio Projection(Α;ψ A ) (2)

[0053] Where χ represents the original audio input, A=[a1,…,a t ,…,a T ]∈R T×1024 , F A =[f1 A ,…,f t A ,…,f TA ]∈R T×d ,θ A and ψ A represents the parameters of the audio encoder and audio feature projection layer, T is the frame length and d is the feature dimension.

[0054] As a preferred embodiment, the pseudo facial key point extractor includes an audio encoder and an Audio2lmk decoder, and finally outputs 3D pseudo facial key points L = [l1, ..., l t ,…,l T ]∈R T×68×3 The audio encoder of the pseudo facial key point extractor shares parameters with the audio encoder in the audio feature extractor. Facial landmarks represent facial expression movements and their dynamics. In order to maintain the unique speaker style, the present invention uses a personalization factor (factor) P∈R N , which is a hot vector of identity labels, is used for personalized facial key point modulation, where N represents the number of subjects used to train DCPTalk. The personalized modulation module contains a linear pseudo facial key point projection layer and a linear personalized factor (factor) projection layer. The pseudo facial key point L in the representation space is modulated by element-wise addition using the personalized factor P. The relevant formula is expressed as follows:

[0055] L = Audio2lmk Decoder(Α;θ L ) (3)

[0056] F L =Personlized Modulation(L,P;ψ L ) (4)

[0057] Among them, F L =[f1 L ,…,f t L ,…,f T L ]∈R T×d is the personalized pseudo facial key point feature, θ L and ψ L They represent the parameters of the Audi o2lmk decoder and the personalized modulation module respectively.

[0058] In order to fuse and align the personalized pseudo facial key point features F L and audio feature F A , the present invention uses a module similar to the Transformer decoder. It includes the steps of: S1a, using a multi-head self-attention layer (ALiBi) with linear bias to LIt assigns higher weights to historical actions that are closer to the current action; S1b uses a multi-head cross attention layer to convert the F obtained by the self-attention layer A and With F L Alignment to promote information interaction and integration between different modes; S1c, after a feed-forward layer, obtains personalized pseudo multimodal features F P =[f1 P ,…,f t P ,…,f T P ]∈R T×d .

[0059] S2: Construct a decoder based on FDCP to decode the F P Features to obtain facial animation. The decoder consists of Mouth Mapping, Mouth2Face and Refine Decoder;

[0060] like Figure 2 As shown on the right side of the figure, the present invention proposes a decoder based on FDCP for decoding the F P The speech-driven facial animation process is divided into three stages: synthesizing mouth movements from driving signals, using mouth movements to evoke facial animation, and refining facial animation. These three stages are implemented by Mouth Mapping, Mouth2Face, and Refine Decoder, respectively. Compared with upper face movements, mouth movements are easier to produce, thanks to their direct connection with speech pronunciation and vocal tract dynamics, so a Mouth Mapping module is constructed; Mouth2Face is the core of the FDCP decoder, which can use mouth movements to evoke facial animation; facial animation involves not only the activation of facial muscles, but also the dynamic changes of surface skin structure, so Refine Decoders are used to further refine vertex-level facial animation (Y M ), thereby obtaining more detailed facial movement information.

[0061] Mouth Mapping is preferably composed of linear layers to synthesize accurate mouth movements. Where V M is the number of vertices in the mouth area. The above process is defined as follows:

[0062]

[0063] Among them, θ M Parameters for the mouth mapping module.

[0064] As a preferred embodiment, Figure 2 As shown in Figure 1, Mouth2Face includes a Transform-based mouth encoder, a Transform-based face decoder, and a set of facial muscle control commands. Based on the prior knowledge of facial animation, the present invention uses the mapping rules between pre-trained mouth movements and facial animation. Specifically, the mouth encoder converts the mouth movement Converted into a signal Q = [q1,…,q t ,…,q T ]∈R T×H×P , where H and P represent the number of commands controlling facial animation and the dimension of each command, respectively. In neural control systems, the interaction of neurons usually relies on electrical impulses to transmit information. By using the element-wise quantization function Discrete facial muscle control signal Q to obtain facial muscle control command Facial muscle control commands are indistinguishable during optimization. Activate relevant facial muscles to achieve facial animation Where V is the number of vertices of the entire face. The formula for this process is as follows:

[0065]

[0066]

[0067]

[0068] Among them, φ M and φ F These are the parameters of the mouth encoder and face encoder, respectively.

[0069] As a preferred method, Mouth2Face uses mouth movements and the facial animation Y=[y1,…,y t ,…,y T ]∈R T×V×3 Conduct training.

[0070] As a preference, Refine Decoder consists of a linear layer based on F P The displacement of each vertex is predicted and then added to the vertex position of the face caused by the mouth movement in a basic way. The refinement stage can be expressed as follows:

[0071]

[0072] in is the facial animation at the refinement stage, φ RIt is the parameter for refining decoding.

[0073] S3: Training the speech-driven 3D facial animation framework DCPTalk, first use the loss function to train the Mouth2Face module to establish the mapping rules between mouth movement and facial animation. Then fix the parameters of the trained Mouth2Face and start training other components of DCPTalk. The loss functions for training Mouth2Face and other components are given respectively.

[0074] As a preferred approach, use the loss function Train the Mouth2Face module, the loss function consists of a self-reconstruction loss and an intermediate command level loss composition:

[0075]

[0076] Among them, the self-reconstruction loss It can be expressed as:

[0077]

[0078] Where Y = [y1,…,y t ,…,y T ]∈R T×V×3 Represents ground truth facial motion.

[0079] Using command level loss To learn the set of facial expression control commands Q. The relevant formulas are defined as follows:

[0080]

[0081] Among them, λ0 is the weighting factor, and sg(·) indicates the stop gradient operation.

[0082] As a preferred method, training other components of DCPTalk includes the steps of: fixing the parameters of Mouth2Face and CNN encoder. The loss function includes reconstruction loss Speed ​​loss and facial keypoint loss The reconstruction loss Used to measure the difference between synthesized facial animations and ground truth, including mouth movements Mouth-induced facial animation and detailed facial animations Speed ​​loss Used to ensure consistency in the timeline, including mouth movements Evoked facial animations and refined facial animations Facial key point loss Get from input audio. DCPTalk The loss function expression is as follows:

[0083]

[0084] Among them, λ lmk is the weighting factor.

[0085] Reconstruction losses It can be expressed by the following formula:

[0086]

[0087] Among them, λ1, λ2, and λ3 are weighting factors.

[0088] Speed ​​loss It can be expressed by the following formula:

[0089]

[0090] Among them, β1, β2, and β3 are weighting factors.

[0091] Facial key point loss It can be expressed by the following formula:

[0092]

[0093] Where L = [l1,…,l t ,…,l T ]∈R T×68×3 Represents the real 3D facial key points extracted by the Openface 2.0 tool.

[0094] S4: Model optimization, introducing BIWI, Multiface and VOCASET to provide comprehensive analysis and optimization for DCPTalk; they contain paired audio-3D mesh sequences of English speech. And VOCA, MeshTalk, FaceFormer, CodeTalker, FaceDiffuse, DiffSpeaker, TalkingStyle and SelfTalk are all compared with the method of the present invention. For some pre-trained models that lack BIWI, VOCASET or Multiface, their official source code is used to retrain their models. For example, MeshTalk needs to be retrained on BIWI and VOCASET.

[0095] As a preference, DCPTalk is trained on a single NVIDIA A100 GPU. The model parameters are iteratively updated using the Adam optimizer with beta1=0.9, beta2=0.999 and a learning rate of 1×10 -4 The feature dimension d is set to 256. For the facial muscle control command, H and P are set to 32 and 128 respectively. For the loss function in all experiments of the present invention, λ0=1.0, λ lmk =1×10 -5 , λ1=0.1, λ2=2.0, λ3=1.0, β1=0.3, β2=4.0 and β3=10.0.

[0096] BIWI includes audio-visual recordings and their corresponding detailed dynamic 3D facial geometry, which were collected from 14 individuals, including 8 women and 6 men, who were asked to verbally express 40 English sentences. Each sentence was recorded in two scenarios: one in a neutral context and the other in an emotional context. The 3D facial geometry sequences were recorded at a frame rate of 25 frames per second, and each geometry sequence consisted of 23,370 vertices. The present invention focuses on FaceFormer, CodeTalker and SelfTalk, and only uses subsets with emotions. Specifically, the dataset is divided into different subsets: BIWI-Train, BIWI-Val and two test subsets - BIWI-Test-A and BIWI-Test-B. These subsets consist of 192 sequences (6 subjects × 32 sentences), 24 sequences (6 subjects × 4 sentences), 24 sequences (6 subjects × 4 sentences) and 32 sequences (8 non-appearing subjects × 4 sentences), respectively.

[0097] VOCASET consists of 480 audio-3D geometry pairs from 12 subjects. Each subject contributes 40 sequences, each with a duration between 3 and 5 seconds, recorded at 60 frames per second. The 3D head geometry is described by 5023 vertices and 9976 faces. For fair comparison, following FaceFormer, CodeTalker, and SelfTalk, the dataset is divided into a training subset (VOCASET-Train) containing 320 sequences (8 subjects × 40 sentences), a validation subset (VOCASET-Val) containing 40 sentences (2 subjects × 20 sentences), and a test subset (VOCASET-Test) containing 40 sentences (2 subjects × 20 sentences).

[0098] Multiface is a new multi-view high-resolution face dataset collected from 13 individuals. Each subject recorded 50 speech-balanced sentences at 30fps. Under uniform illumination, about 150 different camera views were collected per frame and used to reconstruct a 3D head mesh. The 3D head mesh consists of 6172 vertices and 12294 surfaces. For fair comparison, the dataset is divided into different subsets: Multiface-Train, Multiface-Val, Multiface-Test-A, and Multiface-Test-B, referring to the Face Diffuser method. These subsets consist of 360 sequences (9 subjects × 40 sentences), 45 sequences (9 subjects × 5 sentences), 45 sequences (9 subjects × 5 sentences), and 20 sequences (4 subjects × 5 sentences), respectively.

[0099] S5: Quantitative evaluation of the model, based on the methods of FaceFormer, CodeTalker and SelfTalk, evaluates the synchronization between speech content and lip movement by calculating the lip vertex error (LVE), which is the average of the maximum L2 errors induced by the vertices of the lip area in all frames.

[0100] In addition, facial expressions directly reflect the naturalness and liveliness of facial animations, which plays a vital role in enhancing the interactive experience and effectiveness of the proposed method. The upper face dynamic deviation (FDD) metric can be used to measure the upper face dynamic deviations occurring in a series of animations and compare them with the ground truth.

[0101] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A voice-driven facial animation simulation method based on facial muscle linkage, characterized in that: Includes steps: S1: Construct a PPMF encoder, which consists of an audio feature extractor and a pseudo facial key point extractor, and uses a module similar to the Transformer decoder to fuse and align personalized pseudo facial key point features F L and audio feature F A ; S2: Construct a decoder based on FDCP to decode the F P Features are used to obtain facial animation. The decoder consists of Mouth Mapping, Mouth2Face, and Refine Decoder. The three stages of synthesizing mouth movements from driving signals, using mouth movements to evoke facial animation, and refining facial animation are implemented by Mouth Mapping, Mouth2Face, and Refine Decoder respectively. S3: Training the voice-driven 3D face animation framework DCPTalk, first using the loss function The Mouth2Face module is trained to establish the mapping rules between mouth movement and facial animation. Then the parameters of the trained Mouth2Face are fixed and the other components of DCPTalk are trained. The loss functions for training Mouth2Face and other components are given respectively. The loss function includes reconstruction loss Speed ​​loss and facial keypoint loss S4: Model optimization, introducing BIWI, Multiface and VOCASET to provide comprehensive analysis and optimization for DCPTalk, and integrating VOCA, MeshTalk, FaceFormer, CodeTalker, FaceDiffuse, DiffSpeaker, Both TalkingStyle and SelfTalk are compared with the method of the present invention; then DCPTalk is trained on a single NVIDIA A100 GPU; S5: Quantitative evaluation of the model, based on the methods of FaceFormer, CodeTalker and SelfTalk, evaluates the synchronization between speech content and lip movement by calculating the lip vertex error LVE.

2. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 1, characterized in that: In step S1, the pseudo facial key point feature F L and audio feature F A The fusion steps include: S1a: Using a multi-head self-attention layer ALiBi with a linear bias to L to process; S1b: Use a multi-head cross attention layer to convert the F obtained by the self-attention layer A and With F L Alignment; S1c: After the feed-forward layer, the personalized pseudo multimodal feature F is obtained P =[f1 P ,…,f t P ,…,f T P ]∈R T×d .

3. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 2, characterized in that: The audio feature extractor consists of an audio encoder A and an audio feature projection layer F A The composition is expressed as follows: A=Audio Encoder(x;θ) A ) F A =Audio Projection(A;ψ A )。 4. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 2, characterized in that: The pseudo facial key point extractor consists of an audio encoder and an Audio2lmk decoder, which finally outputs 3D pseudo facial key points L = [l1,…,l t ,…,l T ]∈R T×68×3 ; Use the personalization factor P∈R N Personalized facial keypoint modulation; Using the personalization factor P, the pseudo facial key point L in the representation space is modulated by element-wise additive modulation, and the formula is expressed as follows: L=Audio2lmk Decoder(Α;θ L ) F L =Personalized Modulation(L,P;ψ L )。 5. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 1, characterized in that: In step S2, the process of Mouth Mapping for synthesizing accurate mouth movements is expressed by the formula: Mouth2Face uses the mouth encoder to convert mouth movements Converted to a signal Q that controls facial muscle activity by using an element-wise quantization function Discrete facial muscle control signal Q to obtain facial muscle control command Activate relevant facial muscles to achieve facial animation The process is expressed by the formula: Refine Decoder is based on F P The refinement stage predicts the displacement of each vertex and then adds the vertex displacement to the vertex position of the face caused by the mouth movement in a basic way by formulating:

6. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 5, characterized in that: Mouth2Face uses mouth movements separately and the facial animation Y=[y1,…,y t ,…,y T ]∈R T×V×3 Conduct training.

7. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 1, characterized in that: In step S3, the process of DCPTalk training through the loss function includes Mouth2Face module training, which is expressed by the formula:

8. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 1, characterized in that: In step S4, for the lack of pre-trained models of BIWI, VOCASET or Multiface, their official source codes are used to retrain their models.

9. The voice-driven facial animation simulation method based on facial muscle linkage according to claim 1, characterized in that: In step S4, the model parameters are iteratively updated using the Adam optimizer with beta1 = 0.9, beta2 = 0.999 and a learning rate of 1×10 -4 ; The feature dimension d is 256; The facial muscle control commands H and P are set to 32 and 128 respectively; The loss function, λ0=1.0, λ lmk =1×10 -5 , λ1=0.1, λ2=2.0, λ3=1.0, β1=0.3, β2=4.0 and β3=10.0.

Images