A method and system for using audio to drive mimicry of facial expressions

Abstract

This invention relates to a method and system for audio-driven mimicry of facial expressions, comprising the following steps: inputting a speech sequence in any language, a specified emotion to be mimicked by a 3D speaker, and a neutral mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation, to obtain an animated facial expression of the 3D speaker that matches the speech sequence and the specified emotion; and based on the obtained animated facial expression, generalizing it to any character using the same facial expression standard using a retargeting technique. This invention utilizes a deep neural network and supervises the generation of 3D speaker facial expressions based on an emotion model, allowing the 3D speaker to simultaneously lip-sync while the emotion model controllably generates 3D speaker facial expressions. This invention can be widely applied in the field of speech-visual multimodal interaction technology.

Description

Technical Field

[0001] This invention relates to a method and system for using audio to drive mimicry of facial expressions, belonging to the field of speech-visual multimodal interaction technology. Background Technology

[0002] Voice-based natural interaction has been widely used in home, automotive and other fields, and is considered to be the next generation of information distribution portal, which has become the focus of competition among major technology giants in recent years.

[0003] like Figure 1 As shown, pure voice interaction lacks the richness of information presentation, making its integration with screens an inevitable trend. The creation of virtual humans who embody all the "souls of AI," combined with corresponding information display "props," and utilizing screens for natural interaction, is considered the most likely ultimate form of human-computer interaction—evolving from the current human-machine interaction to a human-human interaction model. As an indispensable key, the transformation from text (i.e., the content to be expressed) to the virtual human's facial expressions is one of the foundations for achieving this ultimate form of human-computer interaction. With the maturity of Text-to-Speech (TTS) technology, the pressing issue is how to automatically generate virtual human facial expressions from speech.

[0004] Traditional voice-driven 3D speaker mimicry technology only solves the lip-syncing problem but does not consider emotional controllability, mainly for two reasons:

[0005] 1) Voice-driven 3D speaker mimicry for arbitrary language lip-syncing is not yet mature, so its advanced version, namely the generation of emotional facial expressions, is out of the question.

[0006] 2) Lack of 3D data of three-dimensional speakers under different emotions for training. Summary of the Invention

[0007] To address the aforementioned problems, the purpose of this invention is to provide a method and system for using audio-driven mimicry expressions, which can simultaneously present three-dimensional facial expressions according to specified emotions while solving the lip-syncing problem, thereby achieving controllable generation of vivid facial animations.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for using audio to drive mimicry of facial expressions, comprising the following steps:

[0010] Input the speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation to obtain the facial animation of the 3D speaker that matches the speech sequence and the specified emotion.

[0011] Based on the obtained facial animation, a redirection technique is used to generalize to any character that adopts the same facial expression standard.

[0012] Furthermore, the method of inputting a speech sequence in any language, a specified emotion to be presented by the 3D speaker mimicry, and a neutral mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation to obtain an facial animation of the 3D speaker that matches the speech sequence and the specified emotion includes:

[0013] 3D facial expression capture technology based on 2D video acquires offline training data and trains the constructed deep neural network for facial expression generation.

[0014] After encoding the input speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral mesh of the 3D speaker's face, the input is fed into a trained deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion.

[0015] Furthermore, the method for acquiring offline training data using the 3D facial expression capture technology based on 2D video and training the constructed facial expression generation deep neural network includes:

[0016] Obtain the facial shape basis coefficients and expression basis coefficients for each frame from the 2D video, represented by a 3D deformable model.

[0017] Extracting emotional parameters of the target person from each frame of a 2D video;

[0018] The deep neural network for expression generation is trained based on the obtained facial shape base coefficients, the coefficients of each frame's expression base, and the emotional parameters of the target character.

[0019] Furthermore, the method for obtaining the facial shape basis coefficients and expression basis coefficients of each frame represented by a three-dimensional deformable model from a 2D video includes:

[0020] The 2D video data is structured into 3D audio and video data; the 3D audio and video data includes audio, 3D facial expressions, and the emotions being expressed.

[0021] Based on the obtained 3D audio and video data, the facial shape base coefficients and the expression base coefficients of each frame are obtained by representing the face in a three-dimensional deformable model.

[0022] Furthermore, when obtaining the emotional parameters of the target person in each frame of the 2D video, a typical emotion model or a positive / negative arousal emotion model is used.

[0023] The typical emotion model takes a 2D image of a human face as input and outputs one of the following emotion types: neutral, sad, happy, contemptuous, disgusted, surprised, fearful, and angry, as well as the emotional intensity of that emotion type.

[0024] The positive / negative arousal emotion model takes a 2D image of a facial expression as input and outputs positive / negative values ​​and arousal levels.

[0025] Furthermore, the method of encoding the input speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral mesh of the 3D speaker's face, and then inputting them into a trained deep neural network for expression generation to obtain an animated expression of the 3D speaker that matches the speech sequence and the specified emotion, includes:

[0026] A universal speech encoder is used to convert input speech sequences in any language into speech atom representation sequences;

[0027] The neutral state mesh of the 3D speaking face is converted into the face shape basis coefficients of the 3D deformable model used to obtain the face shape control vector;

[0028] Based on the specified emotional parameters required for the 3D speaker mimicry, an emotion control vector is obtained according to a preset mapping method;

[0029] The speech atom representation sequence, face shape control vector, and emotion control vector are input into the expression generation deep neural network to obtain the facial expression animation of the three-dimensional speaker that matches the speech sequence and the specified emotion.

[0030] Furthermore, the method for converting an input speech sequence in any language into a speech atom representation sequence using a universal speech encoder includes:

[0031] The input speech sequence in any language is divided into several speech segments with a length of 20ms.

[0032] Each speech segment is encoded to obtain the speech atom representation corresponding to that speech segment, thereby encoding the speech sequence into a speech atom representation sequence.

[0033] Furthermore, the deep neural network for facial expression generation includes an encoder unit, a decoder unit, and a model output unit. The encoder unit is used to convert the input speech atom representation sequence, face shape control vector, and emotion control vector into coefficients of the expression basis. The decoder unit contains the expression basis itself and is used to multiply the expression basis coefficients with the expression basis to obtain the facial expression change. The model output unit is used to add the obtained facial expression change to a given 3D speaker face mesh to obtain a frame-by-frame 3D speaker facial expression animation that matches the speech sequence and the specified emotion.

[0034] Furthermore, the encoder unit includes a temporal fusion layer, a first fully connected layer, a normalization network layer, and a second fully connected layer; the input of the temporal fusion layer is a matrix composed of windowed speech atom representation sequences and a conditional vector, which is passed through the first fully connected layer to obtain expression basis coefficients; the normalization network layer is used to normalize the encoded expression basis coefficients and output them through the second fully connected layer.

[0035] In a second aspect, the present invention provides a system for using audio to drive mimicry of facial expressions, comprising:

[0036] The 3D facial expression generation module is used to input a speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation, so as to obtain the facial expression animation of the 3D speaker that matches the speech sequence and the specified emotion.

[0037] The redirection module is used to generalize the obtained facial animation to any character that uses the same facial expression standard, using redirection techniques.

[0038] The present invention has the following advantages due to the adoption of the above technical solutions:

[0039] 1. This invention utilizes a deep neural network for facial expression generation. Based on an emotion model, it performs supervised training to generate 3D speaking facial expressions. While completing lip-syncing, the emotion model controllably generates 3D speaking facial expressions. The neural network is designed to conform to the retargeting standard, enabling it to generalize to any character.

[0040] 2. To further improve the lip-syncing problem for arbitrary languages, this invention first employs a self-supervised pre-training method based on large-scale speech data to train multilingual atomic speech representations. Based on the generalization ability of these atomic representations, fine-tuning training is then performed on a speech-to-3D face pairing dataset to generate 3D speaking facial expressions. Compared to traditional phoneme (or character)-based generation, this generation method truly achieves multilingual representation because phonemes or characters are only effective for specific languages ​​or language families.

[0041] 3. To address the lack of 3D data on speakers in different emotional states, this invention employs emotion extraction technology based on 2D video and 3D facial expression capture technology to automatically extract and classify the speaker's emotions and 3D facial expressions in 2D video, and to structure large-scale 2D facial speech data for use in training this task.

[0042] Therefore, this invention can be widely applied in the field of voice and vision multimodal interaction technology. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0044] Figure 1 This is a background diagram of the development of voice interaction;

[0045] Figure 2 This is a schematic diagram of the method for using audio to drive mimicry of facial expressions provided in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart of a method for using audio-driven mimicry expressions provided in an embodiment of the present invention;

[0047] Figure 4 This is a capture effect diagram of extracting 3D expressions from 2D video provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of emotion extraction results obtained from 2D facial expression images provided in an embodiment of the present invention;

[0049] Figure 6a and Figure 6b These are the positive / negative vulnerabilities-arousal model diagram and its extraction effect diagram provided in the embodiments of the present invention;

[0050] Figure 7 This is an overall framework diagram of the method for using audio-driven mimicry expressions provided in the embodiments of the present invention;

[0051] Figure 8 This is a schematic diagram illustrating the preliminary effect of virtual human voice-driven operation provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0054] like Figure 2 As shown, the core theme of this invention is audio-driven mimicry, which is a technology for generating 3D speaker facial expression animations driven by speech and controllable emotions. The main research content of this invention is how to generate 3D facial expression mimicry (i.e., the rough-scale animation stage) through emotional speech and time-varying emotional state functions, and then generalize the 3D facial expression mimicry to any animated character (i.e., the hyper-realistic virtual human animation stage) through retargeting technology. The emotional speech is synthesized from the time-varying emotional state function and content text based on TTS technology, while the time-varying emotional state function and content text are obtained based on emotional state generation and dialogue content generation. This technology is well-known to those skilled in the art, and this invention will not elaborate further. The solution proposed in this invention allows facial expressions to move with changes in human emotions and supports generation in multiple languages.

[0055] Correspondingly, in other embodiments of the present invention, a system for using audio to drive mimicry of facial expressions is provided.

[0056] Example 1

[0057] like Figure 3 As shown, this embodiment provides a method for using audio to drive mimicry of facial expressions, which includes the following steps:

[0058] 1) Input the speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state (zero-pose) mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation (articulation network) to obtain the facial animation of the 3D speaker that matches the speech sequence and the specified emotion, i.e., the 3D mesh sequence.

[0059] 2) Based on the obtained facial animation, the retargeting technique is used to generalize it to any character that adopts the same facial expression standard.

[0060] Preferably, step 1) above can be implemented in the following way:

[0061] 1.1) Offline training data acquisition: Offline training data is acquired using 3D facial expression capture technology based on 2D video, and the constructed deep neural network for facial expression generation is trained.

[0062] 1.2) Online animation generation: The input speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral mesh of the 3D speaker's face are encoded and then input into a trained deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion.

[0063] Preferably, step 1.1) above can be implemented in the following way:

[0064] 1.1.1) Obtain the facial shape basis coefficients and expression basis coefficients of each frame from the 2D video, represented by a 3D Morphable Model (3DMM).

[0065] 1.1.2) Employ affective computing technology to obtain the affective parameters of the target person in each frame of the 2D video;

[0066] 1.1.3) Based on the obtained facial shape base coefficients and the coefficients of each frame's expression base, as well as the emotional parameters of the target character, the constructed deep neural network for expression generation is trained.

[0067] Preferably, in step 1.1.1) above, as follows: Figure 4 As shown, due to the lack of 3D data of speakers under different emotions and the difficulty in large-scale collection, this embodiment uses 3D expression capture technology based on 2D video to obtain the facial shape base coefficients and the coefficients of each frame's expression base from the 2D video, represented by a 3D Morphable Model (3DMM).

[0068] Specifically, this can be achieved through the following steps:

[0069] This paper employs 3D facial expression capture technology based on 2D video to structure 2D audio and video data into 3D audio and video data. The 3D audio and video data includes audio, 3D facial expressions, and the associated emotions. In this embodiment, 3D facial expression capture technology is used to automatically extract data from 2D videos. This data acquisition mechanism requires no annotation, can acquire data at low cost, and provides a foundation for large-scale training.

[0070] Based on the obtained 3D audio and video data, the facial shape base coefficients and the expression base coefficients of each frame are obtained by representing the face in a three-dimensional deformable model.

[0071] In order to complete the downstream retargeting task, the facial expression base is frozen during the training process in this invention. The facial expression base can be preset using standards such as Arkit, or it can be adapted to various different types of facial expression bases.

[0072] Preferably, the aforementioned 3D facial expression capture technology is DECA (Detailed Expression Capture and Animation).

[0073] Preferably, the above-mentioned three-dimensional deformable model can also be selected from other statistical face models, such as the Basel face model (BFM), according to actual needs.

[0074] Preferably, in step 1.1.2) above, the emotion computing technology used in this embodiment includes two mainstream emotion models: the prototypical emotion model and the valence-arousal emotion model. The prototypical emotion model mainly corresponds to emotion type + emotion intensity, while the valence-arousal emotion model mainly corresponds to valence (positive or negative emotion) and arousal (distance from neutral). Both divide emotion into two dimensions, and the prototypical emotion model is a discrete model, while the valence-arousal emotion model is a continuous model.

[0075] like Figure 5 As shown, for the extraction of discrete typical emotion model parameters, this embodiment uses a deep neural network expression recognition method to train the data source. The input of the model is a 2D image of a facial expression, and the output is one of the emotion types such as neutral, sad, happy, contempt, disgusted, surprised, fear, and angry. Based on the confidence interval, the emotion intensity (weak, medium, strong) is given.

[0076] like Figure 6a and Figure 6b As shown, for a continuous positive / negative arousal emotion model, this embodiment uses deep learning to extract valence and arousal values. The input to this model is a 2D image of a facial expression, and the output is valence and arousal values, which can correspond to the parameter values ​​of a discrete typical emotion model.

[0077] Specifically, in this embodiment, the typical emotion model includes 7*3+1=22 emotion categories (happy*3+sad*3+…+angry*3+neutral=22). These 22 emotions can be encoded by an 11-dimensional one-hot vector, where the first 3 dimensions represent the three emotion intensities and the last 8 dimensions represent the emotion type. For example, the vector [1,0,0,1,0,0,0,0,0,0,0] can be defined to represent happy+strong, where the first 1 represents strong and the fourth 1 represents happy. The Valence-arousal model can be any value within the unit circle shown in the figure above.

[0078] Preferably, in step 1.1.3) above, such as Figure 7 As shown, the deep neural network for facial expression generation constructed in this embodiment adopts an Encoder-Decoder structure, which includes an encoder unit, a decoder unit, and a model output unit. The encoder unit converts the input speech atom representation sequence and conditional vector into coefficients of the facial expression basis. The decoder unit contains the facial expression basis itself and multiplies the coefficients of the facial expression basis with the facial expression basis to obtain the facial expression changes, i.e., the per-vertex displacement of each corner point. The model output unit adds the obtained facial expression changes to a given 3D mesh model of a 3D speaker to obtain a frame-by-frame 3D speaker facial expression animation that matches the speech sequence and the specified emotion.

[0079] In order to complete the downstream retargeting task, the emoji base is frozen during the training process in this embodiment. The emoji base can be preset using standards such as ARKit. This embodiment can be adapted to various types of emoji bases.

[0080] Preferably, the encoder unit includes a temporal fusion layer, a first fully connected (FC) layer, a tanh network layer, and a second fully connected layer. The input to the temporal fusion layer is a matrix composed of windowed speech atom representation sequences and a conditional vector. After passing through the first fully connected layer, it yields the articulation embedding, i.e., the expression basis coefficients. The tanh network layer is a normalization layer used to normalize the encoded expression basis coefficients to the range (-1, 1), and then outputs them through the second fully connected layer. In the matrix composed of speech atom representation sequences, the middle frame is the current frame. The current frame and its neighboring frames together constitute a speech segment representation. In this embodiment, this is set to a length of 16 audio frames, 16 * 20 = 320 ms.

[0081] In the discrete typical emotion model, the output emotion control vector contains 8 emotions and 3 emotion intensities. This is encoded as an 11-dimensional one-hot vector and then fused with the 21-dimensional face shape control vector (21-dimensional, consisting of the first 21 coefficients of the face shape basis derived from a 3D mesh) to obtain a 32-dimensional input vector. In the continuous positive / negative arousal model, the output emotion control vector is continuously encoded in a 2-dimensional valence-arousal manner. This 2-dimensional vector represents the emotion parameters of the positive / negative arousal model. It is then multiplied by the face shape control vector and expanded into a 42x1 vector, used to fuse face shape and emotion information.

[0082] Preferably, the temporal fusion layer includes a first residual module (ResNet Block), a second residual module, a third residual module, and a fourth residual module. The four residual modules constitute a deep neural network to achieve deep fusion of three modal data (face shape + emotion + voice).

[0083] Preferably, step 1.2) above can be implemented in the following way:

[0084] 1.2.1) Speech signal encoding: A generic speech encoder is used to convert the input speech sequence in any language into a sequence of atomic speech representations;

[0085] 1.2.2) Image signal: Using mesh fitting technology, the neutral state mesh of the 3D speaking face is converted into the face shape basis coefficients of the 3D deformable model used to obtain the face shape control vector;

[0086] 1.2.3) Emotional signal: Based on the specified emotional parameters required for the three-dimensional speaker mimicry, an emotion control vector is obtained according to a preset mapping method;

[0087] 1.2.4) Output signal generation: Input the speech atom representation sequence, face shape control vector and emotion control vector into the expression generation deep neural network to obtain the facial expression animation of the three-dimensional speaker that matches the speech sequence and the specified emotion.

[0088] Preferably, in step 1.2.1) above, the general speech encoder is trained using a self-supervised pre-training method with masked-language modeling to obtain multilingual atomic speech representations. Specifically, the conversion method includes:

[0089] The input speech sequence in any language is divided into several speech segments with a length of 20ms.

[0090] Each speech segment is encoded to obtain a high-dimensional speech atom representation corresponding to that speech segment, and then a speech atom representation sequence representing that speech sequence is obtained.

[0091] In this embodiment, each high-dimensional speech atom representation is a word in the codebook, and its dimension is determined according to actual needs. In this embodiment, the dimension of the high-dimensional speech atom representation is determined to be 768 dimensions. For example, assuming that the length of the input speech sequence is 1 second, after the general speech encoder encodes the speech sequence, it will output a speech atom representation sequence consisting of 50 vectors, where the dimension of each vector is 768x1.

[0092] Preferably, in step 1.2.2) above, each 3D mesh model is composed of 3D corner coordinates and surfaces formed by the connections between these corners. The corners and their connections within the 3D mesh model constitute its topology. In this embodiment, the topology cannot be changed; that is, the number of corners, their order, and their connections must all remain unchanged.

[0093] Each face base can be considered as a 3D face deformation with the same topological structure, and can be represented as:

[0094]

[0095] In the formula, S mod S represents the coordinates of the corner points of the face being represented. i For each face shape base, a i Let be the face shape basis coefficients. Assume any human face can be linearly represented by m face shape bases. Typically, the first face shape base is the average face, and each subsequent face shape base is a principal component representing the variation relative to the average face. From a... i The resulting vector is called the face shape control vector or face shape parameter.

[0096] Preferably, in step 2) above, since the deep neural network for facial expression generation constructed in this invention adopts an encoder-decoder structure and the output of the encoder is the facial expression basis coefficients, this invention can generalize the deep neural network model for facial expression generation to any character using the same facial expression basis standard, including but not limited to hyper-realistic virtual humans, cartoon characters, etc.

[0097] Example 2

[0098] Embodiment 1 above provides a method for using audio to drive mimicry of facial expressions. Correspondingly, this embodiment provides a system for using audio to drive mimicry of facial expressions. The system provided in this embodiment can implement the method for using audio to drive mimicry of facial expressions in Embodiment 1. This system can be implemented through software, hardware, or a combination of both. For example, the system may include integrated or separate functional modules or units to execute the corresponding steps in the methods of Embodiment 1. Since the system in this embodiment is basically similar to the method embodiment, the description process in this embodiment is relatively simple. For relevant details, please refer to the description of Embodiment 1. The system embodiment provided in this embodiment is merely illustrative.

[0099] This embodiment provides a system for using audio to drive mimicry of facial expressions, including:

[0100] The 3D facial expression generation module is used to input a speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation, so as to obtain the facial expression animation of the 3D speaker that matches the speech sequence and the specified emotion.

[0101] The redirection module is used to generalize the obtained facial animation to any character that uses the same facial expression standard, using redirection techniques.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for using audio to drive mimicry of facial expressions, characterized in that, Includes the following steps: Input the speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation to obtain the facial animation of the 3D speaker that matches the speech sequence and the specified emotion. Based on the obtained facial animation, a redirection technique is used to generalize it to any character that adopts the same facial expression standard; The method for inputting a speech sequence in any language, a specified emotion to be presented in the 3D speaker mimicry, and a neutral mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion includes: acquiring offline training data using 3D expression capture technology based on 2D video, and training the constructed deep neural network for expression generation; encoding the input speech sequence in any language, the specified emotion to be presented in the 3D speaker mimicry, and the neutral mesh of the 3D speaker's face, and then inputting them into the trained deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion; The method for acquiring offline training data and training the constructed deep neural network for expression generation based on 2D video 3D expression capture technology includes: acquiring the facial shape basis coefficients and the coefficients of each frame's expression basis represented by a three-dimensional deformable model from the 2D video; acquiring the emotional parameters of the target person in each frame of the 2D video; and training the constructed deep neural network for expression generation based on the obtained facial shape basis coefficients, the coefficients of each frame's expression basis, and the emotional parameters of the target person. The method for encoding an input speech sequence in any language, a specified emotion to be presented in the 3D speaker mimicry, and a neutral mesh of the 3D speaker's face, and then inputting these encodings into a trained deep neural network for expression generation to obtain an animated facial expression of the 3D speaker that matches the speech sequence and the specified emotion, includes: using a general speech encoder to convert the input speech sequence in any language into a speech atom representation sequence; converting the neutral mesh of the 3D speaker's face into face shape base coefficients of the 3D deformable model used to obtain a face shape control vector; obtaining an emotion control vector based on the specified emotion parameters to be presented in the 3D speaker mimicry according to a preset mapping method; and inputting the speech atom representation sequence, the face shape control vector, and the emotion control vector into the deep neural network for expression generation to obtain an animated facial expression of the 3D speaker that matches the speech sequence and the specified emotion. The method for converting an input speech sequence in any language into a speech atom representation sequence using a universal speech encoder includes: dividing the input speech sequence in any language into several speech segments of 20ms each; encoding each speech segment to obtain a speech atom representation corresponding to that speech segment, thereby encoding the speech sequence into a speech atom representation sequence.

2. The method for using audio to drive mimicry of facial expressions as described in claim 1, characterized in that, The method for obtaining the facial shape basis coefficients and expression basis coefficients of each frame of a face represented by a three-dimensional deformable model from a 2D video includes: The 2D video data is structured into 3D audio and video data; the 3D audio and video data includes audio, 3D facial expressions, and the emotions being expressed. Based on the obtained 3D audio and video data, the facial shape base coefficients and the expression base coefficients of each frame are obtained by representing the face in a three-dimensional deformable model.

3. The method for using audio to drive mimicry of facial expressions as described in claim 1, characterized in that, When obtaining the emotional parameters of the target person in each frame of the 2D video, a typical emotion model or a positive / negative arousal emotion model is used. The typical emotion model takes a 2D image of a human face as input and outputs one of the following emotion types: neutral, sad, happy, contemptuous, disgusted, surprised, fearful, and angry, as well as the emotional intensity of that emotion type. The positive / negative arousal emotion model takes a 2D image of a facial expression as input and outputs positive / negative values ​​and arousal levels.

4. The method for using audio to drive mimicry of facial expressions as described in claim 1, characterized in that, The facial expression generation deep neural network includes an encoder unit, a decoder unit, and a model output unit. The encoder unit is used to convert the input speech atom representation sequence, face shape control vector, and emotion control vector into coefficients of the expression basis. The decoder unit contains the expression basis itself and is used to multiply the coefficients of the expression basis with the expression basis to obtain the facial expression change. The model output unit is used to add the obtained facial expression change to the neutral state mesh of the given 3D speaker's face to obtain a frame-by-frame 3D speaker facial expression animation that matches the speech sequence and the specified emotion.

5. The method for using audio to drive mimicry of facial expressions as described in claim 4, characterized in that, The encoder unit includes a temporal fusion layer, a first fully connected layer, a normalized network layer, and a second fully connected layer. The input to the temporal fusion layer is a matrix composed of windowed speech atom representation sequences and a conditional vector. After passing through the first fully connected layer, expression basis coefficients are obtained. The normalized network layer is used to normalize the encoded expression basis coefficients and output them through the second fully connected layer.

6. A system for driving mimicry of facial expressions using audio, characterized in that, include: The 3D facial expression generation module is used to input a speech sequence in any language, the specified emotion to be presented by the 3D speaker mimicry, and the neutral state mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation, so as to obtain the facial expression animation of the 3D speaker that matches the speech sequence and the specified emotion. The redirection module is used to generalize the obtained facial animation to any character that uses the same facial expression standard using redirection techniques. The method for inputting a speech sequence in any language, a specified emotion to be presented in the 3D speaker mimicry, and a neutral mesh of the 3D speaker's face into a pre-constructed deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion includes: acquiring offline training data using 3D expression capture technology based on 2D video, and training the constructed deep neural network for expression generation; encoding the input speech sequence in any language, the specified emotion to be presented in the 3D speaker mimicry, and the neutral mesh of the 3D speaker's face, and then inputting them into the trained deep neural network for expression generation to obtain an expression animation of the 3D speaker that matches the speech sequence and the specified emotion; The method for acquiring offline training data and training the constructed deep neural network for expression generation based on 2D video 3D expression capture technology includes: acquiring the facial shape basis coefficients and the coefficients of each frame's expression basis represented by a three-dimensional deformable model from the 2D video; acquiring the emotional parameters of the target person in each frame of the 2D video; and training the constructed deep neural network for expression generation based on the obtained facial shape basis coefficients, the coefficients of each frame's expression basis, and the emotional parameters of the target person. The method for encoding an input speech sequence in any language, a specified emotion to be presented in the 3D speaker mimicry, and a neutral mesh of the 3D speaker's face, and then inputting these encodings into a trained deep neural network for expression generation to obtain an animated facial expression of the 3D speaker that matches the speech sequence and the specified emotion, includes: using a general speech encoder to convert the input speech sequence in any language into a speech atom representation sequence; converting the neutral mesh of the 3D speaker's face into face shape base coefficients of the 3D deformable model used to obtain a face shape control vector; obtaining an emotion control vector based on the specified emotion parameters to be presented in the 3D speaker mimicry according to a preset mapping method; and inputting the speech atom representation sequence, the face shape control vector, and the emotion control vector into the deep neural network for expression generation to obtain an animated facial expression of the 3D speaker that matches the speech sequence and the specified emotion. The method for converting an input speech sequence in any language into a speech atom representation sequence using a universal speech encoder includes: dividing the input speech sequence in any language into several speech segments of 20ms each; encoding each speech segment to obtain a speech atom representation corresponding to that speech segment, thereby encoding the speech sequence into a speech atom representation sequence.

Images