Three-dimensional facial animation generation method and system based on 3DMM

Abstract

The invention relates to the technical field of computer vision, in particular to a three-dimensional facial animation generation method and system based on 3DMM, and the method comprises the steps: constructing a three-dimensional facial model; obtaining a target audio, extracting emotional features in the target audio, and inputting the emotional features into the trained ExpNet network to obtain a predicted expression coefficient; applying the predicted expression coefficient to a three-dimensional face model to generate a facial expression of the three-dimensional face model; and inputting the target audio into a trained PoseVAE network to obtain predicted head posture information, and applying the predicted head posture information to a three-dimensional face model to obtain a head animation. In the facial animation generation process, the internal relation between the audio and the facial expression is fully considered, automatic, high-accuracy and diversified generation effects are achieved through the deep learning model, and more real and natural facial animation experience is provided for a user.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology. More specifically, the present invention relates to a 3DMM-based three-dimensional facial animation generation method and system thereof. Background Art

[0002] As a cutting-edge research field at the intersection of artificial intelligence and computer graphics, 3D facial animation generation technology not only has extraordinary academic value, but also has great potential for practical application. This technology has been applied in many industries such as games and virtual reality. By precisely simulating changes in facial expressions, it gives users a more natural and immersive interactive experience. The requirements and standards for 3D facial animation vary in different occasions. For example, in film production, dubbing, and other scenarios that require virtual agents, the naturalness and realism of 3D facial animation must be extreme. These scenarios set high standards for the clarity and expressiveness of animation, requiring it to accurately capture and display the speaker's emotions and language subtleties to provide a real conversation experience. Speech-driven 3D facial animation synthesis uses a piece of speech audio as a driving signal to control the mouth shape and generate a target face video that matches the given audio. This emerging technology has a wide range of application scenarios, such as movie dubbing, video conferencing, online education, and virtual stand-ins. Although existing methods have made some progress in the problem of speech-driven 3D facial animation, which can initially maintain the consistency of speech and facial movements, how to ensure the movement details when speaking is still the main difficulty. The lack of motion detail results in an average, small-action result that's hard to be visually satisfying.

[0003] The current generation of 3D facial animation mainly relies on complex modeling and manual adjustments, or deep learning-based methods to capture and reconstruct facial features. However, these methods either require a lot of manual participation or lack the accuracy and diversity of facial expressions. In addition, existing technologies rarely consider the direct connection between audio and facial movements, and the generated facial animation still lacks authenticity and naturalness. Summary of the invention

[0004] The present invention provides a 3DMM-based three-dimensional facial animation generation method and system thereof, aiming to solve the problem in the related art that the prior art seldom considers the direct connection between audio and facial movement, and the generated facial animation still lacks authenticity and naturalness.

[0005] In the first aspect, the present invention provides a three-dimensional facial animation generation method based on 3DMM, including: constructing a three-dimensional facial model; obtaining target audio, and extracting emotional features in the target audio, inputting the emotional features into a trained ExpNet network to obtain predicted expression coefficients; applying the predicted expression coefficients to the three-dimensional facial model to generate facial expressions of the three-dimensional facial model; inputting the target audio into a trained PoseVAE network to obtain predicted head posture information, applying the predicted head posture information to the three-dimensional facial model to obtain head animation; generating a final facial animation based on the head animation and facial expressions.

[0006] Furthermore, constructing a three-dimensional face model includes: generating a three-dimensional face model based on 3D video data, wherein the three-dimensional face model includes an initial expression coefficient and initial head posture information.

[0007] Furthermore, obtaining the target audio includes: preprocessing the target audio, wherein the preprocessing includes but is not limited to denoising and standardization.

[0008] Furthermore, extracting emotional features from the target audio includes: extracting features representing emotions from the target audio using an audio feature extraction algorithm as emotional features.

[0009] Furthermore, based on the head animation and facial expressions, a final facial animation is generated, including: mapping the predicted expression coefficients and the predicted head posture information to a three-dimensional key point space of a facial renderer; combining the head animation and facial expressions, and generating the final animation using a three-dimensional facial renderer.

[0010] Further, generating a final facial animation includes: post-processing the generated final facial animation, including color correction, denoising and compression.

[0011] Furthermore, obtaining head animation also includes: integrating the trained PoseVAE network into a real-time system to generate diverse head animations by inputting random noise or specific style guidance.

[0012] Furthermore, features representing emotions are extracted from the target audio using an audio feature extraction algorithm, including: extracting features representing emotions from the target audio using a Mel-frequency cepstral coefficient method.

[0013] In a second aspect of the present invention, a three-dimensional facial animation generation system based on 3DMM is also provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the three-dimensional facial animation generation methods based on 3DMM described above.

[0014] Beneficial effects: In the facial animation generation process, the intrinsic connection between audio and facial expressions is fully considered, and the deep learning model is used to achieve automated, highly accurate and diverse generation effects, providing users with a more realistic and natural facial animation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following detailed description is read with reference to the accompanying drawings, which illustrate several embodiments of the present invention in an exemplary and non-limiting manner, and in which like or corresponding reference numerals represent like or corresponding parts, wherein: Figure 1 is a flowchart schematically illustrating generation of a final facial animation according to an embodiment of the present invention; Figure 2 FIG. 1 is a schematic diagram schematically showing a system structure according to an embodiment of the present invention.

[0016] Reference numerals 1. Processor; 2. Communication bus; 3. Memory; 4. Communication interface. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] The current generation of 3D facial animation mainly relies on complex modeling and manual adjustments, or deep learning-based methods to capture and reconstruct facial features. However, these methods either require a lot of manual participation or lack the accuracy and diversity of facial expressions. In addition, existing technologies rarely consider the direct connection between audio and facial movements, and the generated facial animation still lacks authenticity and naturalness.

[0020] S101: constructing a three-dimensional face model.

[0021] In one embodiment, a three-dimensional face model is constructed, wherein the three-dimensional face model construction includes an initial head posture coefficient and an initial expression coefficient. Specifically, the three-dimensional face model construction includes the steps of data collection and preprocessing, shape modeling, texture modeling, expression modeling, head posture modeling and integrated modeling.

[0022] In one embodiment, data collection and preprocessing include: collecting a large number of 3D face scan data of different individuals, which should contain different expressions and head postures, and then aligning all 3D face scan data into a common reference coordinate system for subsequent processing. Shape modeling includes: using principal component analysis (PCA) to extract the main components of the shape from the aligned 3D face scan data. PCA reduces high-dimensional data to a low-dimensional principal component space, which can capture most shape changes. The shape coefficients (also called shape parameters) obtained by PCA can be used to generate new 3D shapes, and different facial shapes can be generated by adjusting these coefficients. Texture modeling includes: mapping each 3D face scan data into a standard UV coordinate system to generate a texture image. Similar to shape modeling, PCA analysis is performed on the texture image to extract the main components of the texture. The texture coefficients obtained by PCA can be used to generate new texture images, thereby changing the appearance of the face. Expression modeling includes: constructing an expression basis, which is a set of 3D deformations that can capture changes in facial expressions. Different expressions can be generated by linearly combining the deformations in the expression basis and adjusting the expression coefficients. Head pose modeling includes: using rotation matrices and translation vectors to represent the pose of the head. The rotation matrix is ​​used to represent the rotation of the head (such as pitch, yaw, and roll), while the translation vector is used to represent the position of the head. These rotation matrices and translation vectors can be regarded as pose coefficients, and the pose of the head can be changed by adjusting these coefficients. The integrated model includes: integrating the coefficients of shape, texture, expression, and pose into a unified framework, and generating a new 3D face model by linearly combining these coefficients. Given a set of shape coefficients, texture coefficients, expression coefficients, and pose coefficients, a 3D face model with a specific shape, texture, expression, and pose can be generated.

[0023] In one embodiment, principal component analysis is a technique for simplifying a data set. It can map high-dimensional data to a low-dimensional space, and retain the features that contribute the most to the variance in the data set, that is, retain the main features of the data, by linear transformation (orthogonal transformation) of data projection with a smaller data set dimension. Specifically, PCA calculates the covariance matrix of the original data, and then finds the eigenvectors and eigenvalues ​​of the covariance matrix. The eigenvectors are the basis vectors of the new space, and the eigenvalues ​​describe the importance of each eigenvector.

[0024] S102: Obtain predicted expression coefficient.

[0025] In one embodiment, a target audio is obtained, and emotional features in the target audio are extracted, and the emotional features are input into a trained ExpNet network to obtain a predicted expression coefficient, which represents facial expression features, such as the opening and closing of eyes, the degree of opening of the mouth, etc. The predicted expression coefficient is applied to a three-dimensional face model to generate the facial expression of the three-dimensional face model.

[0026] In one embodiment, the emotional features in audio are information related to the speaker's emotions extracted from the sound signal. The emotional features in audio mainly include basic emotions such as happiness, anger, surprise, sadness, etc. These emotions can be expressed through the time structure, amplitude structure, fundamental frequency structure and formant structure of the sound. For example, it can be reflected through pitch, volume, speaking speed and advanced attributes such as the emotional color and emotional intensity of the sound.

[0027] In one embodiment, training the ExpNet network includes: first collecting audio data containing various emotions (such as happiness, sadness, anger, surprise, etc.). These data can come from public emotion databases, such as RAVDESS, CREMA-D, etc. Then prepare three-dimensional face models corresponding to the audio data, which should contain facial expression coefficients, which can be obtained by 3DMM or other facial capture technologies. Then preprocess the audio data, such as denoising, standardization, etc., and use audio feature extraction algorithms (such as MFCC, Chroma features, spectrograms, etc.) to extract features that can represent emotions from the audio as emotional features. The audio features are used as input to the network. The network processes the input features through a series of convolutional layers, pooling layers, fully connected layers, etc., and extracts high-level features that can represent audio emotions. The output layer of the network outputs facial expression coefficients, which are used to drive the three-dimensional face model to generate corresponding facial expressions. Finally, the prepared audio and 3D face data are used for training to ensure that the network can learn the corresponding relationship between audio and facial expression coefficients. At this point, a trained ExpNet network can be obtained.

[0028] In one embodiment, the network structure of ExpNet is relatively complex, but the core part mainly includes an audio encoder and a linear layer. The audio encoder is used to encode audio features and generate expression coefficients; while the linear layer is used to further process these expression coefficients to obtain the final 3D facial expression estimation result. In addition, in the present invention, the ExpNet network is combined with the PoseVAE network to achieve more complex facial expression and head pose estimation tasks, and the specific steps are as follows.

[0029] In one embodiment, it also includes: evaluating whether the generated facial expressions can accurately reflect the emotional changes in the audio through manual evaluation or automatic evaluation methods (such as emotion recognition accuracy, facial expression fidelity, etc.).

[0030] S103: Get head animation.

[0031] In one embodiment, the target audio is input into the trained PoseVAE network to obtain predicted head posture information, which includes the rotation angle (pitch angle, yaw angle and roll angle) of the head and possible translation information; the predicted head posture information is applied to the three-dimensional face model to obtain head animation. It should be noted that the PoseVAE network, that is, the posture variational autoencoder, is a network model specially designed to learn and generate realistic, identifiable, and posture-rich head movement representations from input data. The PoseVAE network has important applications in audio-driven speaking head animation generation. Specifically, it can extract head posture information from the input audio features and generate head movement sequences that match the audio content. This capability makes the PoseVAE network have broad application prospects in virtual character generation, head posture simulation in video calls, and head animation generation in games and animations.

[0032] Specifically, the predicted head posture information is obtained, including: audio preprocessing: first, use an appropriate audio processing library (such as librosa or pydub in Python) to load the target audio file. In one embodiment, the audio can also be preprocessed, including steps such as noise reduction, standardization, and segmentation to ensure the quality of the audio data and make it suitable for input into the PoseVAE network. Extract features from the preprocessed audio to obtain audio features. In this embodiment, the audio features are Mel frequency cepstral coefficients. In other embodiments, the audio features can also be spectrograms, chromaticity graphs, etc., which can be changed according to specific actual scenarios. Use a deep learning framework to load the trained PoseVAE network model. Ensure that the model weights are consistent with those saved during training. Check the input configuration of the PoseVAE network, including the dimensions of the input features, data format, etc., to ensure that the audio features match them. Then input the extracted audio features into the PoseVAE network. Forward propagation is performed through the encoder part of the network to obtain a representation in the latent space. Finally, the decoder part is used to generate the predicted head posture information from the latent space. This usually includes parameters such as rotation matrices, translation vectors, or Euler angles to describe the position and orientation of the head. At this point, the predicted head pose information is parsed from the output of the PoseVAE network.

[0033] S104: Generate a final facial animation based on the head animation and facial expression.

[0034] In one embodiment, the predicted expression coefficient and the predicted head posture information are mapped to the three-dimensional key point space of the facial renderer; the final animation is generated by combining the head animation and facial expression using the three-dimensional facial renderer. Among them, the three-dimensional facial renderer is a tool that uses computer graphics technology to convert a three-dimensional facial model into a realistic two-dimensional image or animation. It usually includes multiple functional modules such as model import, material editing, lighting settings, rendering output, etc., which can simulate the texture, gloss, shadow and other effects of facial skin, thereby generating highly realistic facial images.

[0035] Specifically, facial renderers usually use a set of predefined 3D key points to represent the key features of the face. These key points may include the corners of the eyes, the corners of the mouth, the tip of the nose, etc. The predicted expression coefficients and head posture information are converted into adjustments to these 3D key points. Then, a 3D facial renderer is used for rendering. During the rendering process, the adjusted 3D key point space is used as input, and the head animation (such as the overall movement of the head) and facial expressions (such as the bending of the eyebrows, the opening and closing of the eyes, etc.) are considered at the same time. The 3D key point space is continuously updated, and a series of frames are generated based on these updates to generate the final facial animation.

[0036] In one embodiment, after the final facial animation is generated, some post-processing steps such as smoothing, color correction, and shadow rendering may be required to improve the realism and smoothness of the animation. Smoothing aims to reduce jagged edges and jitters in the animation to make the picture more delicate and smooth; color correction aims to adjust the color balance, brightness, and contrast in the animation to ensure that the animation is visually attractive and consistent with the expected visual effects. Shadow rendering aims to enhance the three-dimensionality and depth of the animation to make the picture more realistic and vivid.

[0037] In one embodiment, during the facial animation generation process, the intrinsic connection between audio and facial expressions is fully considered, and an automated, highly accurate, and diverse generation effect is achieved through a deep learning model, providing users with a more realistic and natural facial animation experience.

[0038] The present invention also provides a three-dimensional facial animation generation system based on 3DMM. Figure 2 As shown, the system includes a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a 3D facial animation generation method based on 3DMM according to the first aspect of the present invention is implemented.

[0039] The system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface, whose configuration and functions are known in the art and thus will not be described in detail here.

[0040] In one embodiment, the present invention provides a computer device. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities, and various varieties such as CPU, single-chip microcomputer, DSP or FPGA can be selected. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. When the computer program is executed, the steps described in the above method embodiment can be completed, such as S101-step S104. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a three-dimensional facial animation generation method based on 3DMM is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0041] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0042] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0043] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0044] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A 3D facial animation generation method based on 3DMM, characterized in that: include: Constructing a 3D facial model; Obtain target audio, extract emotional features from the target audio, input the emotional features into the trained ExpNet network, and obtain predicted expression coefficients; Applying the predicted expression coefficient to a three-dimensional face model to generate a facial expression of the three-dimensional face model; Input the target audio into the trained PoseVAE network to obtain predicted head posture information, and apply the predicted head posture information to the three-dimensional face model to obtain head animation; Based on the head animation and facial expressions, a final facial animation is generated.

2. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Construct a 3D face model, including: Based on the 3D video data, a three-dimensional face model is generated, wherein the three-dimensional face model includes an initial expression coefficient and initial head posture information.

3. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Get the target audio, including: The target audio is preprocessed, wherein the preprocessing includes but is not limited to denoising and normalization.

4. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Extracting emotional features from the target audio, including: An audio feature extraction algorithm is used to extract features representing emotions from the target audio as emotion features.

5. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Based on the head animation and facial expressions, generate final facial animation, including: Mapping the predicted expression coefficient and the predicted head posture information to a three-dimensional key point space of a facial renderer; The head animation and facial expressions are combined to generate a final animation using a three-dimensional facial renderer.

6. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Generate final facial animation, including: The resulting final facial animation was post-processed, including color correction, denoising, and compression.

7. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: Get head animation, also includes: The trained PoseVAE network is integrated into a real-time system to generate diverse head animations by inputting random noise or specific style guidance.

8. The 3DMM-based 3D facial animation generation method according to claim 1, characterized in that: The features representing the emotions are extracted from the target audio using an audio feature extraction algorithm, including: The features representing the emotions are extracted from the target audio using the Mel-frequency cepstral coefficient method.

9. A 3D facial animation generation system based on 3DMM, comprising a processor and a memory, characterized in that: The memory stores a computer program, and the processor executes the computer program to implement the 3DMM-based three-dimensional facial animation generation method according to any one of claims 1 to 8.

Images