A voice-driven method for generating facial expression animations for 3D virtual avatars

Abstract

This invention relates to a method for generating facial expression animations of a voice-driven 3D virtual avatar, comprising: inputting a speech segment and converting it into audio features; an encoder module encoding the audio features extracted by the audio processing module, a static 3D head mesh template, and the input one-hot encoding of the speaking style, encoding them into the latent space, and concatenating them; processing the concatenated feature vector sequence through a bidirectional Mamba module, capturing contextual information for each feature vector position, and outputting a feature vector sequence after information exchange of the same dimension; a decoder decoding the feature vector sequence output by the bidirectional Mamba module to obtain the vertex position offsets of each frame of the face mesh, adding them to the static 3D head mesh template to obtain a 3D head mesh sequence; and using the audio-head mesh sequence dataset to perform end-to-end training of the entire network model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, graphics, and deep learning, and specifically to a method for generating voice-driven facial expression animations of 3D virtual avatars based on Mamba technology. Background Technology

[0002] Audio-driven 3D facial animation generation can be applied in film, animation production, and games. Lip-sync facial animation can enhance the user experience in virtual reality and augmented reality, leading to greater immersion. It can also be used to create virtual digital humans, such as virtual voice assistants and virtual e-commerce livestreamers. Most existing methods are based on CNN or Transformer architectures. CNNs require progressively increasing the receptive field, which doesn't perform well when generalizing to long speech sequences. While Transformers can achieve good results, their computational complexity increases quadratically with sequence length, requiring significant computational resources for long audio files. Summary of the Invention

[0003] This invention addresses the shortcomings of existing voice-driven 3D virtual avatar facial expression animation generation technologies by providing a method for voice-driven 3D virtual avatar facial expression animation generation based on Mamba technology. This method accepts speech segments, static 3D head mesh templates, and speaking styles as input, and outputs a lip-synchronized 3D head mesh sequence corresponding to the speaking style and the speech signal. When accepting long audio as input, this method achieves better lip-synchronization in the output, and the computational complexity increases linearly with the audio length.

[0004] To achieve the above objectives, the present invention employs the following technical solutions.

[0005] A method for generating facial expression animations of a voice-driven 3D virtual avatar includes the following steps:

[0006] Step 1: Input a speech segment. The speech segment is converted into audio features by the audio processing module inside the network framework.

[0007] Step 2: The encoder module encodes the audio features extracted by the audio processing module, the static 3D head mesh template, and the input speech style one-hot code, encoding them into the latent space and concatenating them. The concatenated feature vector sequence is represented as follows: Where L represents the sequence length and D represents the feature vector dimension;

[0008] Step 3: Process the concatenated feature vector sequence using the bidirectional Mamba module. For each feature vector at a given location, contextual information is captured, and a sequence of feature vectors with the same dimension after information exchange is output. This sequence of feature vectors after information exchange is represented as β. 1:T The bidirectional Mamba module exchanges α via three independent paths. 1:T Information between sequences at different positions in the input sequence: The first path captures information forward; the second path captures information backward by reversing the input sequence along the time dimension, and then reverses the feature vector sequence back along the time dimension after capture; the third path is a skip connection that directly passes the input information to the result; the results of the first and second paths are weighted and summed, and then multiplied element-wise with the result of the third path to obtain the final result. The bidirectional Mamba module structure outputs the feature vector sequence as follows:

[0009] Step 4: The decoder processes the feature vector sequence β output by the bidirectional Mamba module. 1:T Decode the face mesh of each frame to obtain the vertex position offset, and add it to the static 3D head mesh template to obtain the 3D head mesh sequence.

[0010] Step 5: Complete the mapping from audio to 3D head mesh sequence through the above steps, and use the audio-head mesh sequence dataset to train the entire network model end-to-end.

[0011] Furthermore, in step 1, the audio processing module uses the deep learning method wav2vec 2.0 to extract speech features. After the temporal convolutional layer in the wav2vec 2.0 model, a linear interpolation layer is added to interpolate the audio features into the fps value of the corresponding grid sequence, so as to achieve the alignment of the audio and grid sequence and facilitate training.

[0012] Furthermore, the method in step 2 is as follows: The encoder module consists of three parts: an audio encoder, a mesh encoder, and a style encoder. The audio encoder is used to encode audio features into the latent space while compressing the feature dimension; the mesh encoder, based on speech style, encodes the input static 3D head mesh template into the latent space, captures facial features, and is used to assist in generating a mesh sequence with a certain speech style.

[0013] Furthermore, in step 2, the static 3D head mesh template is represented using the FLAME model.

[0014] Furthermore, in step 2, linear layers are used as the implementation method for each encoder to encode each part into the latent space. The encoded grid feature vector and speech style feature vector are then copied and expanded along the time dimension, and the three feature vector sequences are concatenated along the time dimension.

[0015] Furthermore, in step 4, a multilayer perceptron is used to implement the decoder module, which outputs the offset of each vertex in each frame relative to the static 3D head mesh. This offset is added to the static 3D head mesh template to obtain the complete 3D head mesh sequence generated by the model.

[0016] Furthermore, in step 5, the audio-head mesh sequence dataset used for training contains paired audio and three-dimensional head mesh sequences, and includes multiple speaking styles.

[0017] Furthermore, in step 5, the loss function used to train the model includes three terms: reconstruction loss L. rec Speed ​​loss L vel and lip loss L lip The loss function for training the model is L = 1 × L rec +1×L vel +0.3×L lip . Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a voice-driven facial animation generation method for a three-dimensional virtual character based on the Mamba model, as described in an example of the present invention.

[0020] Figure 2 This is a schematic diagram of the overall network structure of the method of the present invention.

[0021] Figure 3 This is a schematic diagram of the bidirectional Mamba module structure proposed in the method of this invention.

[0022] Figure 4 This diagram illustrates the generation results of the same word by different individuals and speaking styles according to the present invention.

[0023] Figure 5 This is a comparison of the method of this invention with other methods in the field on the test sets of the two public datasets VOCASET and BIWI using two evaluation metrics: LVE (Lip Vertex Error) and FDD (Upper-Face Dynamics Deviation). Detailed Implementation

[0024] To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of the present invention, the technology in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of the present invention.

[0025] This invention proposes a method for generating facial expression animations of 3D virtual avatars based on the Mamba model, which includes the following steps:

[0026] Step 101: Input a speech segment. The speech segment is processed by the audio processing module inside the network framework and converted into audio features, which are presented in the form of a feature vector sequence.

[0027] It should be noted that the audio processing module converts the original input audio into audio features. In specific implementations, the audio processing module can adopt various methods, such as manual extraction and the use of machine learning models, and this invention does not limit this. The audio features output by the module can be of various types, such as temporal and frequency domain features, but they should meet the actual needs of subsequent model training and analysis, preserving the key information in the speech to the greatest extent possible.

[0028] As a preferred approach, the deep learning method wav2vec 2.0 is used as the audio processing module to extract speech features. Since the audio sampling rate in wav2vec 2.0 is 50Hz, a linear interpolation layer is added after the temporal convolutional layer in the wav2vec 2.0 model to interpolate the 50Hz audio features to the corresponding sequence's fps value (30fps or 60fps), aligning the audio and grid sequence for easier training.

[0029] Step 102: The encoder module encodes the audio features extracted by the audio processing module, the static 3D head mesh template, and the input speech style one-hot code, encoding them into the latent space and concatenating them. The concatenated feature vector sequence is represented as follows: Where L represents the sequence length and D represents the feature vector dimension.

[0030] It should be noted that the encoder comprises three parts: an audio encoder, a mesh encoder, and a style encoder. The audio encoder encodes audio features into the latent space while compressing the feature dimensionality. The mesh encoder, based on the fact that speech style can be represented to some extent by facial vertex positions, encodes the input static 3D head mesh template into the latent space, capturing facial features to assist in generating a mesh sequence with a specific speech style. The representation of the input static 3D head mesh template depends on the dataset and can be represented in various ways (hybrid shape model, parametric 3D head model, etc.), which this invention does not limit. The style encoder encodes the one-hot encoding representing the speech style into the latent space. This invention does not limit the specific implementation of each encoder.

[0031] As a preferred approach, the static 3D head mesh template is represented using the FLAME model (FacesLearned with an Articulated Model and Expressions, FLAME).

[0032] As a preferred approach, linear layers are used as the implementation method for each encoder, encoding each part into the latent space. The encoded grid feature vector and speech style feature vector are then copied and expanded along the time dimension, and the three feature vector sequences are concatenated along the time dimension.

[0033] Step 103: Process the spliced ​​α using a bidirectional Mamba module. 1:T For each feature vector, rich contextual information is captured, and a sequence of feature vectors with the same dimension after information exchange is output. This sequence of feature vectors after information exchange is represented as β. 1:T .

[0034] It should be noted that this invention does not limit the specific implementation of the bidirectional Mamba module, but it should be designed based on the Mamba structure and should enable the feature vector at a certain position to be aware of the context in both the forward and backward directions.

[0035] As a preferred embodiment, this invention provides a design method for implementing a bidirectional Mamba module. The feature vector sequence after the encoder... It can represent all the information input from each part. For example... Figure 3 As shown, the bidirectional Mamba module exchanges α via three independent paths. 1:TInformation between sequences at different positions in the input. The first path is responsible for capturing information forward; the second path captures information backward by reversing the input sequence along the time dimension, and then reverses the feature vector sequence back along the time dimension after capture; the third path is a skip connection, directly passing the input information to the result. This invention example designs a bidirectional Mamba module based on the original Mamba block, which adds a second path compared to the original Mamba block. After obtaining the results of the three paths, the results of the first and second paths are weighted and summed, and then multiplied element-wise with the result of the third path to obtain the final result. The output feature vector sequence of the bidirectional Mamba module structure is...

[0036] Step 104: The decoder processes the β output via the bidirectional Mamba module. 1:T Decode the data to obtain the vertex offsets of each frame of the face mesh, and add them to the static 3D head mesh template to obtain the 3D head mesh sequence.

[0037] It should be noted that the function of the decoder is to map the latent space feature vector sequence into a head vertex displacement sequence. This invention does not restrict the implementation of the decoder module, but the output vertex displacement sequence should be able to reflect the input audio pronunciation features and speaking style.

[0038] As a preferred embodiment, this invention uses a multilayer perceptron to implement the decoder module, outputting the offset of each vertex in each frame relative to the static 3D head mesh. This offset is then added to the static template to obtain the complete 3D head mesh sequence generated by the model.

[0039] Step 105: Complete the mapping from audio to the 3D head mesh sequence through the above steps, and use the audio-head mesh sequence dataset to train the entire network model end-to-end.

[0040] It should be noted that this invention does not limit the dataset used, but the dataset should contain paired audio and three-dimensional head grid sequences, and contain multiple speakers, i.e., multiple speaking styles.

[0041] As a preferred approach, the model is trained using the training sets of the VOCASET and BIWI datasets. The loss function used to train the model consists of three terms: reconstruction loss L... rec Speed ​​loss L vel and lip loss L lip The calculation methods for each loss are as follows:

[0042]

[0043] Where T represents the length of the sequence, V represents the number of vertices in the 3D head mesh, and Vlip The y value represents the number of vertices in the lip region of the 3D head mesh. t,v This represents the true value of the v-th vertex at position t in the sequence. This is the predicted value of the corresponding model. The loss function for the final trained model is L = 1 × L rec +1×L vel +0.3×L lip .

[0044] The results obtained using the above method are as follows Figure 4 As shown.

[0045] Step 106: Evaluate the method proposed in this invention on the test set of the dataset using appropriate evaluation metrics, and compare it with other methods in the art.

[0046] It should be noted that this invention does not impose any restrictions on the evaluation indicators used.

[0047] As a preferred approach, the method of this invention is compared with other methods on the VOCASET and BIWI test sets using LVE (Lip Vertex Error) and FDD (Upper-Face Dynamics Deviation). VOCA, FaceFormer, CodeTalker, and SelfTalk methods are all speech-driven methods for generating 3D head mesh model sequences. The calculation methods for LVE and FDD are as follows:

[0048]

[0049] Where V lip This represents the set of vertices in the lip region. represents the predicted lip vertex position by the model, and v represents the actual position of the corresponding vertex in the dataset.

[0050]

[0051] Where V up This represents the set of vertices in the upper half of the face region. Represents the actual sequence of vertex positions. This represents the predicted position sequence of the same vertex, where dyn represents the standard deviation of the sequence.

[0052] Using the above metrics and datasets, the results compared with other methods are as follows: Figure 5 As shown, the lower the value, the better the effect.

[0053] The above-described working mode is for illustrative purposes only and is not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

Claims

1. A method for generating facial expression animations of a voice-driven 3D virtual avatar, comprising the following steps: Step 1: Input a speech segment. The speech segment is converted into audio features by the audio processing module inside the network framework. Step 2: The encoder module encodes the audio features extracted by the audio processing module, the static 3D head mesh template, and the input speech style one-hot code, encoding them into the latent space and concatenating them. The concatenated feature vector sequence is represented as follows: Where L represents the sequence length and D represents the feature vector dimension; Step 3: Process the concatenated feature vector sequence using the bidirectional Mamba module. For each feature vector at a given location, contextual information is captured, and a sequence of feature vectors with the same dimension after information exchange is output. This sequence of feature vectors after information exchange is represented as β. 1:T The bidirectional Mamba module exchanges α via three independent paths. 1:T Information between sequences at different positions in the input sequence: The first path captures information forward; the second path captures information backward by reversing the input sequence along the time dimension, and then reverses the feature vector sequence back along the time dimension after capture; the third path is a skip connection that directly passes the input information to the result; the results of the first and second paths are weighted and summed, and then multiplied element-wise with the result of the third path to obtain the final result. The bidirectional Mamba module structure outputs the feature vector sequence as follows: Step 4: The decoder processes the feature vector sequence β output by the bidirectional Mamba module. 1:T Decode the face mesh of each frame to obtain the vertex position offset, and add it to the static 3D head mesh template to obtain the 3D head mesh sequence. Step 5: Complete the mapping from audio to 3D head mesh sequence through the above steps, and use the audio-head mesh sequence dataset to train the entire network model end-to-end.

2. The voice-driven three-dimensional virtual character facial expression animation generation method according to claim 1, characterized in that, In step 1, the audio processing module uses the deep learning method wav2vec 2.0 to extract speech features. After the temporal convolutional layer in the wav2vec 2.0 model, a linear interpolation layer is added to interpolate the audio features into the fps value of the corresponding grid sequence, so as to achieve the alignment of the audio and grid sequence and facilitate training.

3. The voice-driven three-dimensional virtual character facial expression animation generation method according to claim 1, characterized in that, Step 2 is as follows: The encoder module consists of three parts: an audio encoder, a mesh encoder, and a style encoder. The audio encoder is used to encode audio features into the latent space while compressing the feature dimension. The mesh encoder, based on the speaking style, encodes the input static 3D head mesh template into the latent space, captures facial features, and is used to assist in generating a mesh sequence with a certain speaking style.

4. The voice-driven three-dimensional virtual character facial expression animation generation method according to claim 1, characterized in that, In step 2, the static 3D head mesh template is represented using the FLAME model.

5. The method for generating facial expression animation of a voice-driven three-dimensional virtual character according to claim 1, characterized in that, In step 2, a linear layer is used as the implementation method for each encoder. Each part is encoded into the latent space, and the encoded grid feature vector and speech style feature vector are copied and extended along the time dimension. Then, the three feature vector sequences are concatenated along the time dimension.

6. The method for generating facial expression animation of a voice-driven three-dimensional virtual character according to claim 1, characterized in that, In step 4, a multilayer perceptron is used to implement the decoder module, which outputs the offset of each vertex of each frame relative to the static 3D head mesh. This offset is then added to the static 3D head mesh template to obtain the complete 3D head mesh sequence generated by the model.

7. The method for generating facial expression animation of a voice-driven three-dimensional virtual character according to claim 1, characterized in that, In step 5, the audio-head mesh sequence dataset used for training contains paired audio and 3D head mesh sequences, and includes multiple speaking styles.

8. The method for generating facial expression animation of a voice-driven three-dimensional virtual character according to claim 1, characterized in that, In step 5, the loss function used to train the model includes three terms: reconstruction loss L. rec Speed ​​loss L vel and lip loss L lip The loss function for training the model is L = 1 × L rec +1×L vel +0.3×L lip .

Images