A method for generating speech-driven 3D facial motion animation based on linguistic keyframes

Abstract

The present invention discloses a method for generating a speech-driven three-dimensional facial motion animation based on linguistic keyframes, comprising the following steps: obtaining a one-dimensional speech signal; preprocessing the one-dimensional speech signal to obtain a keyframe time index; generating keyframe facial motion using the one-dimensional speech signal and the keyframe time index; generating a complete facial motion sequence using the one-dimensional speech signal, the keyframe time index, and the keyframe facial motion; significantly improving the accuracy and naturalness of facial animation, achieving a high degree of synchronization between facial expressions and speech signals, and effectively solving the problem of overly smooth facial motion animation generated by the existing method, especially the problem of missing dynamic details when processing complex facial expressions (such as pouting or opening the mouth). Not only can the generation quality of three-dimensional facial animation be improved, and it is suitable for applications such as virtual reality and augmented reality, but it can also enhance the naturalness and user experience of human-computer interaction.

Description

Technical Field

[0001] The invention relates to the technical fields of computer graphics and artificial intelligence, and in particular to a method for generating a speech-driven three-dimensional human face motion animation based on linguistic key frames. Background Art

[0002] With the rapid development of virtual reality, filmmaking, computer games, and education, generating 3D facial animation synchronized with input speech has become an important technology. As an important innovation in the fields of human-computer interaction and virtual reality, people can drive 3D virtual facial movements in real time through voice signals to achieve more natural communication and interaction. Therefore, in order to enhance the immersion and user experience of human-computer interaction, achieving high-precision synchronization and expression of voice-driven 3D faces is an important step in modern and intelligent human-computer interaction systems. It is of great significance to develop a voice-driven 3D face technology that can be practically applied.

[0003] Speech-driven 3D facial animation generation has its own specific technical difficulties. Generating 3D facial animation directly from speech has the uncertainty of cross-modal mapping. Due to the complex many-to-many mapping relationship between speech and facial expressions, this process often leads to the problem of over-smoothing of the generated results, lacking facial dynamic details and precise lip synchronization.

[0004] Although existing linguistics-based methods can provide fine control, they are difficult to fully cover all possible facial movements due to their reliance on manually designed mapping rules. Although data-driven methods have made significant progress in speech-driven 3D facial animation generation, they often fail to capture detailed facial dynamics when dealing with complex facial expressions (such as pouting or opening the mouth). In particular, when changes in facial expressions need to be highly synchronized with changes in phonemes, the performance of existing methods usually tends to be overly smoothed. Therefore, in order to reduce uncertainty in the cross-modal mapping process and improve the accuracy and naturalness of generated animations, it is of great significance to study and develop a speech-driven 3D facial motion animation generation method based on linguistic keyframes. This will not only improve the user experience in applications such as virtual reality and augmented reality, but will also promote the development of human-computer interaction technology. Summary of the invention

[0005] (1) Technical issues to be resolved

[0006] The present invention discloses a method for generating voice-driven three-dimensional facial motion animation based on linguistic keyframes, aiming to solve the problem of over-smoothing in the difficult voice-driven three-dimensional facial animation generation. A progressive learning mechanism based on keyframe embedding is proposed, which reduces the uncertainty and learning complexity of cross-modal mapping by introducing keyframe capture, thereby generating more vivid, accurate and time-coherent three-dimensional facial animation.

[0007] (2) Technical solution

[0008] The present invention discloses a method for generating a speech-driven three-dimensional face motion animation based on linguistic key frames, comprising the following steps:

[0009] Step 1, obtaining a one-dimensional speech signal;

[0010] Step 2, preprocessing the one-dimensional speech signal to obtain a key frame time index;

[0011] Step 3, using the one-dimensional speech signal and the key frame time index to generate key frame face motion;

[0012] Step 4: Generate a complete face motion sequence using the one-dimensional speech signal, key frame time index, and key frame face motion.

[0013] Further, in step 2, preprocessing the one-dimensional speech signal in step 1 to obtain a key frame time index includes the following steps:

[0014] Step 201, extracting text content from input speech using an automatic speech recognition (ASR) model;

[0015] Step 202, using the Montreal Forced Aligner technology to align the audio with the text, and locate the timestamps of the start and end of the phonemes;

[0016] Step 203, converting these time stamps into corresponding motion frame indexes as key frame time indexes;

[0017] Further, in step 3, the key frame face motion is generated using the one-dimensional speech signal and the key frame time index in step 1, including the following steps:

[0018] Step 301, extracting audio features using the one-dimensional speech signal in step 1;

[0019] Step 302: Generate key frame face motion by using the audio features and the key frame time index through a key frame decoder.

[0020] Further, in step 4, a complete facial motion sequence is generated using the one-dimensional speech signal, key frame time index, and key frame facial motion in step 1, including the following steps:

[0021] Step 401, using a key frame encoder to encode key frame face motion to obtain key frame facial motion features;

[0022] Step 402, using the audio features and the key frame facial motion features to perform feature fusion to obtain multimodal fusion features;

[0023] Step 403, using the multimodal fusion features, decoding is performed through a motion decoder to obtain a temporally complete facial motion sequence.

[0024] Specifically, the process of converting these time stamps into corresponding motion frame indexes as key frame time indexes in step 203 is as follows:

[0025]

[0026] in, are the timestamps of the start and end of the phoneme obtained in step 202, The frame rate at which the video runs for the preset;

[0027] Specifically, in step 301, the audio features are extracted using the one-dimensional speech signal in step 1, using the audio features obtained by the pre-trained wav2vec2.0 network, and then according to the total time length of the audio and the preset frame rate of the video running The number of frames obtained , linearly interpolate the audio features to obtain audio features that match the number of frames:

[0028]

[0029] in It is audio data, specifically a one-dimensional speech signal with a sampling rate of 16kHZ. The audio features are obtained through the language pre-training network wav2vec2.0. is the audio feature that matches the number of frames, where d is the dimension of the audio feature.

[0030] Specifically, the key frame decoder in step 302 uses an improved Transformer network to decode key frame facial motion, and its model is designed to focus on the screening of key frame audio features to adapt to key frame facial motion generation. The key frame decoder network algorithm flow includes the following steps:

[0031] First, enter the time index of the keyframe Perform feature filtering on audio features, specifically, based on the key frame time index The frame index in the filter selects the key frame audio features of the corresponding index:

[0032]

[0033] in, is the encoded audio feature, is the key frame time index;

[0034] Secondly, the key frame audio features are used to decode the key frame decoder based on Transformer to output the predicted key frame facial motion sequence. , where m is the key frame time index The length of is the number of key frame face movements, is the number of vertices of the face mesh.

[0035] In particular, the training process of the Transformer-based key frame decoder introduces a complete facial motion sequence for supervision. Since the key frame facial motion sequence lacks contextual information of adjacent facial motions, if the key frame facial motion sequence is directly regressed, inaccurate facial motion will result. To solve this problem, this method introduces a complete facial motion sequence to supervise the key frame facial motion sequence to ensure that the sequence is generated accurately and naturally. The specific algorithm is as follows:

[0036] First, the time-complete face motion pseudo sequence is generated by using the keyframe face motion sequence and non-keyframe index:

[0037]

[0038] in, is the key frame time index, is the time index of the non-keyframe, is the real face motion sequence label, is the predicted key frame face motion sequence, To pass and Construct a time-complete pseudo-sequence.

[0039] The loss function is then used to minimize the gap between the temporally complete pseudo sequence and the true label:

[0040]

[0041] in, It is the loss function for temporally complete face motion sequence;

[0042] Specifically, It is composed of reconstruction loss function, speed loss function, feature space consistency loss function, and text consistency loss function. Feature space consistency loss and text consistency loss are conducive to the accuracy of lip shape:

[0043]

[0044] in is the offset distance loss function, n is the number of video frames, is the speed loss function, is the feature space consistency loss function, is the text consistency loss function, for After the encoder's features, for The features encoded by the encoder, for and The vector of for The text probability vector output by the prediction head, for The text probability vector output by the prediction head, is the loss function weight;

[0045] Specifically, in step 401, a key frame human face motion is encoded by using a key frame encoder to obtain a key frame facial motion feature. The key frame encoder here uses a key frame encoder based on a flow model for encoding. The specific algorithm flow includes the following steps:

[0046] First, the predicted key frame face motion sequence Encode and generate key frame face motion sequence context tags :

[0047]

[0048] in, represents the multi-head attention module, It consists of two linear layers and an activation function. Contextual labeling for key-frame face motion sequences;

[0049] Next, index the non-keyframes Sinusoidal position encoding is performed to represent non-keyframe positions. Cross-attention layers are used to combine keyframe face motion sequence context tags. and non-keyframe position encoding, from which intermediate motion flow markers are extracted :

[0050]

[0051] in represents sinusoidal position encoding, represents the attention mechanism, , , Represents the QKV matrix weight in cross attention;

[0052] Finally, we use the key frame time index in the implicit space of the explicit motion flow Splicing and The information is used to fuse the feature context through the one-dimensional convolution layer and the Transformer layer to obtain the overall motion flow features. .

[0053] Specifically, in step 402, the feature fusion is performed using the audio feature and the key frame facial motion feature to obtain the multimodal fusion feature. The algorithm flow of the feature fusion is as follows:

[0054] First, the feature fusion weight matrix is ​​learned through the overall motion flow features and audio features through the linear layer and Sigmoid function:

[0055]

[0056] in, is the learned feature fusion weight matrix, is the overall motion flow characteristic, is the audio feature;

[0057] Then, based on the learned weight matrix, the motion flow features and audio features are fused by adaptive gating:

[0058]

[0059] in, is the overall motion flow characteristic, is the audio feature, and Z is the fusion feature obtained by feature gating.

[0060] The present invention discloses a method for generating a speech-driven three-dimensional facial motion animation based on linguistic keyframes, comprising: an audio input module, an audio preprocessing module, a keyframe facial motion generation module and a complete facial motion animation generation module. The audio input module is used to receive a speech signal; the audio preprocessing module uses linguistic priors to identify key phonemes from the speech signal and find the corresponding keyframe index; the audio preprocessing module is used to generate keyframe facial motion through audio and keyframe indexes to generate a preliminary animation; the complete facial motion animation generation module uses keyframe facial motion and audio features to complete the generation of three-dimensional facial animation and ensure a high degree of synchronization between animation and speech. The method of the present invention can improve the naturalness and accuracy of three-dimensional facial animation, and is suitable for virtual reality, augmented reality and other application fields that require speech-driven three-dimensional animation. Beneficial Effects

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention proposes a method for generating three-dimensional facial motion animation driven by voice based on linguistic keyframes. The present invention uses linguistic prior information to accurately extract the first and last frames of phonemes from speech signals as keyframes, and generates corresponding keyframes through audio, which significantly improves the accuracy and naturalness of facial animation. The collaborative work of the keyframe facial motion generation module and the complete facial motion animation generation module achieves a high degree of synchronization between facial expressions and speech signals, effectively solving the problem of overly smooth facial motion animation generated by existing methods, and the problem of missing dynamic details when processing complex facial expressions (such as pouting or opening the mouth). The present invention can not only improve the generation quality of three-dimensional facial animation, and is suitable for applications such as virtual reality and augmented reality, but also improve the naturalness of human-computer interaction and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the present invention;

[0064] Figure 2 The process flow chart of preprocessing the one-dimensional speech signal to obtain the key frame time index in step 2 of the present invention;

[0065] Figure 3 The present invention provides a flowchart for generating a complete facial motion sequence using a one-dimensional speech signal, a key frame time index, and a key frame facial motion in step 4. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] For the purpose of reference and clarity, the technical terms, abbreviations or acronyms used below are summarized and explained as follows:

[0068] Linear: A fully connected layer in a neural network.

[0069] Sigmoid: An activation function that maps input to values ​​between 0 and 1.

[0070] Concat: A vector concatenation operation.

[0071] Transformer: An attention-based neural network architecture for processing sequence data.

[0072] Attention: The attention mechanism in neural networks allows the model to focus on the importance of different parts when processing the input.

[0073] SinPositionEmbedding: A positional encoding method that uses sine and cosine functions to generate unique embeddings for each position in a sequence, helping the model capture sequential information.

[0074] The first purpose of the present invention is to provide a method for generating a speech-driven three-dimensional facial motion animation based on linguistic key frames.

[0075] refer to Figure 1 ;

[0076] A method for generating a speech-driven three-dimensional facial motion animation based on linguistic key frames comprises the following steps:

[0077] Step 1: Get a one-dimensional speech signal ;

[0078] Step 2: One-dimensional speech signal Perform preprocessing to obtain the key frame time index ;

[0079] Step 3: Using one-dimensional speech signal and keyframe time index Perform key frame face motion generation;

[0080] Step 4: Using one-dimensional speech signal , keyframe time index , key frame face motion generates a complete face motion sequence.

[0081] The following is a detailed description of the process of a method for generating a speech-driven three-dimensional facial motion animation based on linguistic key frames with examples.

[0082] Step 1: Get a one-dimensional speech signal Specifically, it can be collected by a collection device and processed by an upstream algorithm to obtain a one-dimensional speech signal.

[0083] Step 2: For the one-dimensional speech signal in step 1 Perform preprocessing to obtain the key frame time index ,refer to Figure 2 ;

[0084] Specifically, it includes the following algorithm flow:

[0085] Step 201, extracting text content from input speech using an automatic speech recognition (ASR) model;

[0086] Step 202, using the Montreal Forced Aligner technology to align the audio with the text, and locate the timestamps of the start and end of the phonemes;

[0087] Step 203, converting these time stamps into corresponding motion frame indexes as key frame time indexes;

[0088] Specifically, in step 203, these timestamps are converted into corresponding motion frame indexes as key frame time indexes. The process is:

[0089]

[0090] in, are the timestamps of the start and end of the phoneme obtained in step 202, The frame rate at which the video runs for the preset;

[0091] In step 3, the one-dimensional speech signal in step 1 is used and the keyframe time index obtained in step 2 The key frame face motion generation includes the following steps:

[0092] Step 301, using the one-dimensional speech signal in step 1 Extract audio features;

[0093] Specifically, the one-dimensional speech signal in step 301 is used as To extract audio features, we use the pre-trained wav2vec2.0 network to get the audio features, and then use the total length of the audio and the preset frame rate of the video to extract the audio features. The number of frames obtained , linearly interpolate the audio features to obtain audio features that match the number of frames:

[0094]

[0095] in is the one-dimensional speech signal in step 1, with a sampling rate of 16 kHz. The audio features are obtained through the language pre-training network wav2vec2.0 , is the audio feature that matches the frame number , where d is the dimension of the audio feature.

[0096] Step 302, using audio features and keyframe time index Generate keyframe face motion through keyframe decoder.

[0097] Specifically, the key frame decoder in step 302 uses an improved Transformer network to decode key frame facial motion, and its model is designed to focus on the screening of key frame audio features to adapt to key frame facial motion generation. The key frame decoder network algorithm flow includes the following steps:

[0098] First, enter the keyframe time index The audio features are filtered. Specifically, the key frame audio features of the corresponding index are filtered out according to the frame index in the key frame time index:

[0099]

[0100] in, is the encoded audio feature, is the key frame time index, is the key frame audio feature;

[0101] Secondly, using keyframe audio features , decoded by the Transformer-based key frame decoder, and outputting the predicted key frame face motion sequence , where m is the key frame time index The length of is the number of key frame face movements, is the number of vertices of the face mesh.

[0102] In particular, the training process of the Transformer-based key frame decoder introduces a complete facial motion sequence for supervision. Since the key frame facial motion sequence lacks contextual information of adjacent facial motions, if the key frame facial motion sequence is directly regressed, inaccurate facial motion will result. To solve this problem, this method introduces a complete facial motion sequence to supervise the key frame facial motion sequence to ensure that the sequence is generated accurately and naturally. The specific algorithm is as follows:

[0103] First, the time-complete face motion pseudo sequence is generated by using the keyframe face motion sequence and non-keyframe index:

[0104]

[0105] in, is the key frame time index, is the time index of the non-keyframe, is the real face motion sequence label, is the predicted key frame face motion sequence, To pass and Construct a time-complete pseudo-sequence.

[0106] The loss function is then used to minimize the gap between the temporally complete pseudo sequence and the true label:

[0107]

[0108] in, It is the loss function for temporally complete face motion sequence;

[0109] Specifically, It is composed of reconstruction loss function, speed loss function, feature space consistency loss function, and text consistency loss function. The feature space consistency loss function and text consistency loss function are conducive to the accuracy of lip shape:

[0110]

[0111] in is the offset distance loss function, n is the number of video frames, is the speed loss function, is the feature space consistency loss function, is the text consistency loss function, for After the encoder's features, for The features encoded by the encoder, for and The vector of for The text probability vector output by the prediction head, for The text probability vector output by the prediction head, is the loss function weight;

[0112] Preferably, , , , .

[0113] In step 4: using the one-dimensional speech signal, the key frame time index, and the key frame face motion to generate a complete face motion sequence, the specific steps include the following steps, see Figure 3 ;

[0114] Step 401, using a key frame encoder to encode the key frame face motion to obtain the key frame facial motion features. The key frame encoder here uses a key frame encoder based on a flow model for encoding. The specific algorithm flow includes the following steps:

[0115] First, the predicted key frame face motion sequence Encode and generate key frame face motion sequence context tags :

[0116]

[0117] in, represents the multi-head attention module, It consists of two linear layers and an activation function. Contextual labeling for key-frame face motion sequences;

[0118] Next, index the non-keyframes Sinusoidal position encoding is performed to represent non-keyframe positions. Cross-attention layers are used to combine keyframe face motion sequence context tags. and non-keyframe position encoding, from which intermediate motion flow markers are extracted :

[0119]

[0120] in represents sinusoidal position encoding, represents the attention mechanism, , , Represents the QKV matrix weight in cross attention;

[0121] Finally, we use the key frame time index in the implicit space of the explicit motion flow Splicing and The information is used to fuse the feature context through the one-dimensional convolution layer and the Transformer layer to obtain the overall motion flow features. .

[0122] Step 402, the audio features and the key frame facial motion features are used to perform feature fusion to obtain multimodal fusion features. The algorithm flow of feature fusion is as follows:

[0123] First, the feature fusion weight matrix is ​​learned through the overall motion flow features and audio features through the linear layer and Sigmoid function:

[0124]

[0125] in, is the learned weight matrix, is the overall motion flow characteristic, is the audio feature;

[0126] Then, based on the learned weight matrix, the motion flow features and audio features are fused by adaptive gating:

[0127]

[0128] in, is the overall motion flow characteristic, is the audio feature, and Z is the fusion feature obtained by feature gating.

[0129] Step 403, using the multimodal fusion features, decoding is performed through a motion decoder to obtain a temporally complete facial motion sequence, wherein the motion decoder is a decoder of a Transformer architecture.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0131] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] The technical features of the above embodiments can be combined arbitrarily. At the same time, the number of each step is not used to constrain the order of the steps. As long as there is no strict order constraint between the steps, the order is allowed to be adjusted and changed. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating a speech-driven three-dimensional facial motion animation based on linguistic key frames, characterized in that: The following steps are involved: Step 1, obtaining a one-dimensional speech signal; Step 2, preprocessing the one-dimensional speech signal to obtain a key frame time index; Step 3, using the one-dimensional speech signal and the key frame time index to generate key frame face motion; Step 4, using the one-dimensional speech signal, the key frame time index, and the key frame face motion to generate a complete face motion sequence; In step 3, the key frame face motion generation is performed using the one-dimensional speech signal and the key frame time index, including the following steps: Step 301, extracting audio features using the one-dimensional speech signal in step 1; Step 302, using the audio features and the key frame time index to generate a key frame face motion sequence through a Transformer-based key frame decoder; In decoding through the Transformer-based keyframe decoder, the training process of training the Transformer-based keyframe decoder introduces the complete face motion sequence for supervision, including the following steps: First, the time-complete face motion pseudo sequence is generated by using the keyframe face motion sequence and the non-keyframe time index: ； in, is the time index of the non-keyframe, is the key frame time index, Expressed as A real face motion sequence label, is the predicted key frame face motion sequence, is the real face motion sequence label, A time-complete pseudo sequence constructed by predicting key-frame face motion sequences and non-key-frame true face motion sequence labels; The loss function is then used to minimize the gap between the temporally complete pseudo sequence and the true label: ； in, is the face motion sequence loss function for the complete sequence supervision.

2. The method for generating a speech-driven 3D facial motion animation based on linguistic key frames according to claim 1, characterized in that: In the step 2, the one-dimensional speech signal is preprocessed to obtain a key frame time index, which includes the following steps: Step 201, extracting text content from input speech using an automatic speech recognition model; Step 202, using the Montreal forced alignment technique to align the audio with the text, and locate the timestamps of the start and end of the phonemes; Step 203: convert these time stamps into corresponding motion frame indexes as key frame time indexes.

3. The method for generating a speech-driven 3D facial motion animation based on linguistic key frames according to claim 1, characterized in that: In step 4, a complete facial motion sequence is generated using a one-dimensional speech signal, a key frame time index, and a key frame facial motion, and the following steps are included: Step 401, using a key frame encoder to encode key frame face motion to obtain key frame facial motion features; Step 402, using the audio features and the key frame facial motion features to perform feature fusion to obtain multimodal fusion features; Step 403, using the multimodal fusion features, decoding is performed through a motion decoder to obtain a temporally complete facial motion sequence.

4. The method for generating a speech-driven 3D facial motion animation based on linguistic key frames according to claim 1, characterized in that: The step 302 of generating key frame face motion by using the audio features and the key frame time index through a key frame decoder includes the following steps: First, input the key frame time index to filter the audio features. Specifically, filter the key frame audio features of the corresponding index according to the key frame time index: ； in, is the audio feature extracted in step 3, is the audio feature at the key frame position, Expressed as The encoded audio features, is the key frame time index obtained in step 2; Secondly, the key frame audio features are used to decode the key frame decoder based on Transformer to output the predicted key frame facial motion sequence. , where m is the key frame time index The length of is the number of key frame face movements, is the number of vertices of the face mesh.

5. The method for generating a speech-driven 3D facial motion animation based on linguistic key frames according to claim 3, characterized in that: The step 401 of encoding the key frame face motion using the key frame encoder to obtain the key frame facial motion features includes the following steps: First, the predicted key frame face motion sequence Encode and generate key frame face motion sequence context tags : ； in, represents the multi-head attention module, It consists of two linear layers and an activation function. Contextual labeling for key-frame face motion sequences; Next, index the non-keyframes Perform sinusoidal position encoding to represent non-keyframe positions, use cross-attention layers, and combine keyframe face motion sequence context tags and non-keyframe position encoding, from which intermediate motion flow markers are extracted : ； in represents sinusoidal position encoding, Represented as a cross attention mechanism, , , represents the QKV matrix weight in the cross attention, Represented as a sinusoidal position encoding of non-keyframes, Expressed as the total number of frames, Represented as a keyframe time index The length of is the number of key frame face movements, Represented as the dimension of audio features; Finally, we use the key frame time index in the implicit space of the explicit motion flow Splicing and The information is used to fuse the feature context through the one-dimensional convolution layer and the Transformer layer to obtain the overall motion flow features. .

6. The method for generating a speech-driven 3D facial motion animation based on linguistic key frames according to claim 3, characterized in that: The step 402 of using the audio features and the key frame facial motion features to perform feature fusion to obtain the multimodal fusion features includes the following steps: First, the feature fusion weight matrix is ​​learned through the overall motion flow features and audio features through the linear layer and Sigmoid function: ； in, is the learned feature fusion weight matrix, is the overall motion flow characteristic, is the audio feature, Represented as a fully connected layer in a neural network, Represented as a vector concatenation operation; Then, based on the learned weight matrix, the motion flow features and audio features are fused by adaptive gating: ； in, is the overall motion flow characteristic, is the audio feature, and Z is the fusion feature obtained by feature gating.

Images