Voice-driven multi-expression 3D facial animation generation method and system

Abstract

The invention discloses a voice-driven multi-expression 3D facial animation generation method and system, and relates to the field of computer animation and virtual reality. The method comprises the following steps: acquiring an input audio, and encoding the audio by using a fine-tuned encoder to obtain features of the input audio; performing feature dimension reduction by using a full-connection neural network to obtain potential codes of audio features; inputting the potential audio code, the emotion code and the diffusion step number into a face denoising model to generate a potential face motion code; performing discrete quantization processing on the facial motion codes by using a codebook to obtain approximate potential facial motion codes in the codebook; and decoding the discretized potential facial motion code to obtain facial motion, and adding the facial motion to a static facial template to obtain a final complete facial motion sequence. According to the method, the 3D facial animation which is high in quality and closely synchronous with the input audio and the emotion label is generated, and a new effective means is provided for creating a virtual character with rich emotion.

Description

Technical Field

[0001] The invention belongs to the technical field of computer animation and virtual reality, and in particular to a method and system for generating multi-expression 3D facial animation driven by speech. Background Art

[0002] 3D facial animation plays a vital role in digital content creation and virtual interaction. Especially in movies, video games, and real-time communications, high-quality facial animation can greatly enhance the audience's immersion and emotional engagement. Traditional 3D facial animation technology often relies on a complex manual modeling process, in which artists must carefully adjust the movement of facial muscles to realistically reproduce human expressions. This process is not only time-consuming and labor-intensive, but also requires a high degree of professional skills.

[0003] With the advancement of artificial intelligence technology, especially the development of deep learning, automated facial animation generation methods have become a hot topic of research. Voice-driven 3D facial animation systems, which use voice input to automatically generate expressions, provide an efficient and dynamic solution. Such systems usually drive the movement of facial models by analyzing the emotions and intonation in voice signals, thereby generating expression animations that match the voice content.

[0004] Emotional facial animation generation further considers the emotional state contained in the speech, aiming to create more delicate and diverse expression changes. For example, the emotional state of anger, happiness or sadness can be identified by analyzing the rhythm, volume and pitch of the speech and reflected in the 3D facial animation. This requires the system to not only accurately recognize the text information in the speech, but also capture the speaker's emotional intentions and convert them into corresponding facial expressions.

[0005] Current challenges in this technology field include how to improve the accuracy of emotion recognition, how to generate natural and smooth facial expressions that match individual characteristics, and how to reduce the demand for computing resources to achieve real-time animation generation.

[0006] Therefore, it is an urgent problem for those skilled in the art to propose a voice-driven multi-expression 3D facial animation generation method and system to solve the difficulties existing in the prior art. Summary of the invention

[0007] The present invention proposes a speech-driven multi-expression 3D facial animation generation method and system, which realizes direct conversion from audio to facial animation by comprehensively using deep learning and diffusion models, generates high-quality 3D facial animation that is closely synchronized with input audio and emotion labels, and provides a new and effective means for creating emotionally rich virtual characters.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A method for generating multi-expression 3D facial animation driven by speech comprises the following steps:

[0010] S1: Get the input audio, first encode the audio using the fine-tuned HuBERT encoder to obtain the features of the input audio; then use a fully connected neural network to perform feature dimensionality reduction to obtain the latent code of the audio features;

[0011] S2: Get the input emotion label provided in the one-hot format and encode it using a fully connected neural network to obtain a latent code containing emotion features;

[0012] S3: Input the latent code containing the audio feature, the latent code containing the emotional feature, and the diffusion step number into the facial denoising model, and generate the latent facial motion code corresponding to the audio feature code and the emotional feature code through the diffusion process;

[0013] S4: The facial motion code is discretely quantized using a codebook to obtain the closest potential facial motion code in the codebook;

[0014] S5: Input the closest potential facial motion code in the codebook into the facial motion decoder for decoding to obtain facial motion, and finally add the current facial motion to the static facial template to obtain the final complete facial motion sequence.

[0015] In the above method, optionally, the implementation of the fully connected neural network in S1 and S2 is expressed as:

[0016] MLP(A)=LN(Mish(LN(A)))

[0017] Among them, A represents the input audio features or emotional features, LN() represents the linear layer, and Mish() represents the mish activation function.

[0018] In the above method, optionally, the emotion labels obtained in S2 in a one-hot format include anger, contempt, disgust, fear, happiness, sadness, and surprise.

[0019] The above method is optional. The face denoising model in S3 consists of an attention mechanism and a fully connected layer, which is expressed as follows:

[0020] FDM A ,F E ,T)=CA(SA(f(MLP(F E ),LN(z t ))),f(LN(T),MLP(F A)))

[0021] Among them, F A F stands for audio potential code; E represents the potential code of emotion; T represents the total number of diffusion steps; CA and SA represent cross attention and self-attention operations respectively; LN represents the linear layer; MLP is a fully connected neural network.

[0022] In the above method, optionally, the discrete quantization processing in S4 is expressed as:

[0023]

[0024] in, represents the discretized potential facial motion; l represents the original potential facial motion; C is the codebook; argmin represents obtaining the closest feature.

[0025] In the above method, optionally, the facial motion decoder in S5 is composed of a convolutional network, a fully connected network and a Transformer network.

[0026] A voice-driven multi-expression 3D facial animation generation system, which executes any of the above-mentioned voice-driven multi-expression 3D facial animation generation methods, comprises an acquisition module, an encoding module, a denoising module, a discrete quantization module, and a decoding module connected in sequence;

[0027] The encoding module includes: a speech encoding module and an emotion encoding module;

[0028] Acquisition module: used to obtain input audio and emotion labels;

[0029] Speech coding module: used to encode audio information into potential audio codes;

[0030] Emotion encoding module: used to encode emotion labels into potential emotion codes;

[0031] Denoising module: used to fuse the latent audio code and the latent emotion code, and generate the latent facial motion code through the diffusion process;

[0032] Discrete quantization module: used to perform discrete quantization processing on the potential facial motion code to find the closest potential facial motion code from the codebook;

[0033] Decoding module: used to decode the discrete quantized facial latent code to obtain the final facial motion sequence.

[0034] A computer storable medium stores a computer program, which, when executed by a processor, implements a voice-driven multi-expression 3D facial animation generation method as described in any one of the above.

[0035] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a voice-driven multi-expression 3D facial animation generation method and system, which has the following beneficial effects:

[0036] 1) Direct conversion from audio to facial animation by combining deep learning and diffusion models;

[0037] 2) The method first encodes the input audio through the HuBERT encoder and then further processes it using a fully connected neural network to obtain the potential audio code; the input emotion label is converted into a potential emotion code through one-hot encoding and neural network processing; these two codes are input into the face denoising model (FDM) together with the determined number of diffusion steps;

[0038] 3) In this model, audio features and emotional features are combined through a diffusion process to generate corresponding latent facial motion codes, which are then discretely quantized through a codebook to determine the closest facial motion code.

[0039] 4) These discretized codes are input into the facial motion decoder and decoded into specific facial movements. These facial movements are combined with a static facial template to generate the final complete facial motion sequence;

[0040] 5) Through the present invention, high-quality 3D facial animations that are closely synchronized with input audio and emotion labels can be generated, providing a new and effective means for creating emotionally rich virtual characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A flow chart of a method for generating multi-expression 3D facial animation driven by speech disclosed in the present invention;

[0043] Figure 2 A stylized animation schematic diagram disclosed in the present invention;

[0044] Figure 3 This is a structural block diagram of a voice-driven multi-expression 3D facial animation generation system disclosed in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] Reference Figure 1 As shown, a speech-driven multi-expression 3D facial animation generation method comprises the following steps:

[0048] S1: Get the input audio, first encode the audio using the fine-tuned HuBERT encoder to obtain the features of the input audio; then use a fully connected neural network to perform feature dimensionality reduction to obtain the latent code of the audio features;

[0049] S2: Get the input emotion label provided in the one-hot format and encode it using a fully connected neural network to obtain a latent code containing emotion features;

[0050] S3: Input the latent code containing the audio feature, the latent code containing the emotional feature, and the diffusion step number into the facial denoising model, and generate the latent facial motion code corresponding to the audio feature code and the emotional feature code through the diffusion process;

[0051] S4: The facial motion code is discretely quantized using a codebook to obtain the closest potential facial motion code in the codebook;

[0052] S5: Input the closest potential facial motion code in the codebook into the facial motion decoder for decoding to obtain facial motion, and finally add the current facial motion to the static facial template to obtain the final complete facial motion sequence.

[0053] Furthermore, the implementation of the fully connected neural network in S1 and S2 is expressed as:

[0054] MLP(A)=LN(Mish(LN(A)))

[0055] Among them, A represents the input audio features or emotional features, LN() represents the linear layer, and Mish() represents the mish activation function.

[0056] In the above method, optionally, the emotion labels obtained in S2 in a one-hot format include anger, contempt, disgust, fear, happiness, sadness, and surprise.

[0057] Furthermore, the face denoising model in S3 consists of an attention mechanism and a fully connected layer, as shown below:

[0058] FDM A ,F E ,T)=CA(SA(f(MLP(F E ),LN(z t ))),f(LN(T),MLP(F A )))

[0059] Among them, F A F stands for audio potential code; E represents the potential code of emotion; T represents the total number of diffusion steps; CA and SA represent cross attention and self-attention operations respectively; LN represents the linear layer; MLP is a fully connected neural network.

[0060] Furthermore, the discrete quantization processing in S4 is expressed as:

[0061]

[0062] in, represents the discretized potential facial motion; l represents the original potential facial motion; C is the codebook; argmin represents obtaining the closest feature.

[0063] Furthermore, the facial motion decoder in S5 is composed of a convolutional network, a fully connected network, and a Transformer network.

[0064] Specific: such as Figure 2 As shown, in order to compare the present invention with other high-performance human shape estimation methods in a quantitative way, the present invention shows the performance of the most advanced methods: FaceFormer, CodeTalker, FaceDiffuser. The data and results are shown in Table 1;

[0065] Table 1: Evaluation of lip error rate, emotion average error rate and face error rate

[0066] method LVE EME FVE FaceFormer 2.6694 3.1015 2.6740 CodeTalker 2.3541 4.2054 2.3901 FaceDiffuser 2.1660 2.6846 2.1519 Ours 2.0211 2.5885 2.0144

[0067] Reference Figure 3 As shown, a voice-driven multi-expression 3D facial animation generation system executes any of the above-mentioned voice-driven multi-expression 3D facial animation generation methods, including an acquisition module, an encoding module, a denoising module, a discrete quantization module, and a decoding module connected in sequence;

[0068] The encoding module includes: a speech encoding module and an emotion encoding module;

[0069] Acquisition module: used to obtain input audio and emotion labels;

[0070] Speech coding module: used to encode audio information into potential audio codes;

[0071] Emotion encoding module: used to encode emotion labels into potential emotion codes;

[0072] Denoising module: used to fuse the latent audio code and the latent emotion code, and generate the latent facial motion code through the diffusion process;

[0073] Discrete quantization module: used to perform discrete quantization processing on the potential facial motion code to find the closest potential facial motion code from the codebook;

[0074] Decoding module: used to decode the discrete quantized facial latent code to obtain the final facial motion sequence.

[0075] A computer storable medium stores a computer program, which, when executed by a processor, implements a voice-driven multi-expression 3D facial animation generation method as described in any one of the above.

[0076] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0077] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating multi-expression 3D facial animation driven by speech, characterized in that: The following steps are involved: S1: Get the input audio, first encode the audio using the fine-tuned HuBERT encoder to obtain the features of the input audio; Next, a fully connected neural network is used to perform feature dimensionality reduction to obtain the latent code of the audio features; S2: Get the input emotion label provided in the one-hot format and encode it using a fully connected neural network to obtain a latent code containing emotion features; S3: Input the latent code containing the audio feature, the latent code containing the emotional feature, and the diffusion step number into the facial denoising model, and generate the latent facial motion code corresponding to the audio feature code and the emotional feature code through the diffusion process; S4: The facial motion code is discretely quantized using a codebook to obtain the closest potential facial motion code in the codebook; S5: Input the closest potential facial motion code in the codebook into the facial motion decoder for decoding to obtain facial motion, and finally add the current facial motion to the static facial template to obtain the final complete facial motion sequence.

2. The method for generating multi-expression 3D facial animation driven by speech according to claim 1, characterized in that: The implementation of the fully connected neural network in S1 and S2 is expressed as: MLP(A)=LN(Mish(LN(A))) Among them, A represents the input audio features or emotional features, LN() represents the linear layer, and Mish() represents the mish activation function.

3. The method for generating multi-expression 3D facial animation driven by speech according to claim 1, characterized in that: The emotion labels obtained in S2 in the one-hot format include anger, contempt, disgust, fear, happiness, sadness, and surprise.

4. The method for generating multi-expression 3D facial animation driven by speech according to claim 1, characterized in that: The face denoising model in S3 consists of an attention mechanism and a fully connected layer, as shown below: FDM(F A ,F E ,T)=CA(SA(f(MLP(F E ),LN(zt))),f(LN(T),MLP(F A ))) Among them, F A F stands for audio potential code; E represents the potential code of emotion; T represents the total number of diffusion steps; CA and SA represent cross attention and self-attention operations respectively; LN represents the linear layer; MLP is a fully connected neural network.

5. The method for generating multi-expression 3D facial animation driven by speech according to claim 1, characterized in that: The discrete quantization processing in S4 is expressed as: in, represents the discretized potential facial motion; l represents the original potential facial motion; C is the codebook; argmin represents obtaining the closest feature.

6. The method for generating a voice-driven multi-expression 3D facial animation according to claim 1, characterized in that: The facial motion decoder in S5 is composed of a convolutional network, a fully connected network, and a Transformer network.

7. A speech-driven multi-expression 3D facial animation generation system, characterized in that: A method for generating a multi-expression 3D facial animation driven by speech according to any one of claims 1 to 6, comprising an acquisition module, an encoding module, a denoising module, a discrete quantization module, and a decoding module connected in sequence; The encoding module includes: a speech encoding module and an emotion encoding module; Acquisition module: used to obtain input audio and emotion labels; Speech coding module: used to encode audio information into potential audio codes; Emotion encoding module: used to encode emotion labels into potential emotion codes; Denoising module: used to fuse the latent audio code and the latent emotion code, and generate the latent facial motion code through the diffusion process; Discrete quantization module: used to perform discrete quantization processing on the potential facial motion code to find the closest potential facial motion code from the codebook; Decoding module: used to decode the discrete quantized facial latent code to obtain the final facial motion sequence.

8. A computer storable medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, a voice-driven multi-expression 3D facial animation generation method as described in any one of claims 1 to 6 is implemented.

Images