Network training method and device, and method and device for driving facial animation by voice

Abstract

The invention relates to a network training method and device, a method and device for driving facial animation through voice, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a target frame image and corresponding audio data from sample video data; recognizing from the target frame image to obtain a face image, performing image processing on the face image through an expression prediction network to obtain a target face motion vector corresponding to the face image, and mapping the target face motion vector into a target mixed shape coefficient by adopting a target mapping matrix; inputting the audio data into the initial mixed shape coefficient prediction network to obtain a predicted mixed shape coefficient corresponding to the audio data; and training the initial mixed shape coefficient prediction network based on the difference between the target mixed shape coefficient and the prediction mixed shape coefficient to obtain a mixed shape coefficient prediction network. By adopting the method, the cost can be reduced while the animation driving effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a network training method and apparatus, a method and apparatus for voice-driven facial animation, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Speech-driven 3D digital human facial animation technology is an advanced computer graphics application that drives the facial expressions and lip shapes of virtual characters by analyzing speech signals. Speech-driven 3D digital human facial animation technology usually combines knowledge from multiple fields such as speech recognition, facial modeling, and animation generation. This technology captures speech input and then converts it into semantic information corresponding to 3D facial animation, such as phonemes, emotion tags, or emotional curves, through speech recognition technology, so that it matches the preset facial movements. Finally, through 3D modeling and animation technology, realistic facial expressions and lip shapes synchronized with speech are generated, allowing digital humans to communicate in a natural and smooth way.

[0003] There are currently two methods for voice-driven 3D digital human emotional facial animation. The first is to use the mixed shape coefficients of the corresponding facial action units annotated by professional artists and pre-made emotional morphological keys, train the deep network model using the annotated data, and use the trained model to infer the mixed shape coefficients. This method has a high production cost. The second is to introduce a differentiable rendering pipeline, reconstruct the three-dimensional deformable model and render the corresponding facial image, and obtain a network model that can be used to infer the three-dimensional deformable model through training with multiple loss functions at the facial key points, pixels, and perception levels. However, the three-dimensional deformable expression coefficients of this method cannot be adapted to the facial binding created by the artist.

[0004] That is to say, the current voice-driven 3D digital human facial animation technology is limited by many issues, such as the lack of 3D facial animation data sets, the inconsistent standards for blended shape morphological keys corresponding to facial action units, and the high randomness and difficulty in predicting facial movements. It has the problems of high cost and poor driving effect. Summary of the invention

[0005] Based on this, it is necessary to provide a network training method and device, a method and device for voice-driven facial animation, a computer device, a computer-readable storage medium and a computer program product that can reduce the cost of voice-driven 3D digital human facial animation while improving the driving effect in response to the above technical problems.

[0006] In a first aspect, the present application provides a network training method, the method comprising:

[0007] Obtaining a target frame image and corresponding audio data from sample video data;

[0008] Recognize a face image from the target frame image, perform image processing on the face image through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and map the target facial motion vector into a target mixed shape coefficient using a target mapping matrix;

[0009] Inputting the audio data into an initial mixed shape coefficient prediction network to obtain predicted mixed shape coefficients corresponding to the audio data;

[0010] Based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient, the initial mixed shape coefficient prediction network is trained to obtain a mixed shape coefficient prediction network.

[0011] In one embodiment, the initial mixed shape coefficient prediction network includes an audio feature extraction module and the prediction module, and the step of inputting the audio data into the initial mixed shape coefficient prediction network to obtain the predicted mixed shape coefficient corresponding to the audio data includes:

[0012] Obtaining the autocorrelation coefficient of the audio data using linear predictive coding;

[0013] Inputting the autocorrelation coefficient and the user-defined emotion intensity into the audio feature extraction module to obtain the emotion feature and speech content feature corresponding to the audio data;

[0014] The prediction module performs prediction processing on the emotion feature and the speech content feature to obtain the prediction mixed shape coefficient corresponding to the audio data.

[0015] In one embodiment, the method further comprises:

[0016] Using the neutral face model, extracting a 51-dimensional first blend shape morphological key set and a 53-dimensional target motion vector set;

[0017] Reconstructing a first 3D face model based on the 51-dimensional mixed shape morphological key set, and reconstructing a second 3D face model based on the 53-dimensional target motion vector set;

[0018] The target position of the second three-dimensional face model is mapped to the first three-dimensional face model through an initial mapping matrix to obtain a mapping mixed shape morphological coefficient, and according to the difference between the mapping mixed shape morphological coefficient and the mixed shape morphological coefficient corresponding to the target position in the first three-dimensional face model, the initial mapping matrix is ​​adjusted to obtain a target mapping matrix.

[0019] In one embodiment, the predicting module predicts the emotion feature and the speech content feature to obtain the predicted mixed shape coefficient corresponding to the audio data, including:

[0020] Taking the emotion feature and the speech content feature as a first sample pair;

[0021] Determining an emotion label corresponding to the emotion feature, and determining a speech content corresponding to the speech content feature;

[0022] Acquire an associated emotional feature corresponding to the emotional tag from the sample data, determine the speech content feature corresponding to the associated emotional feature and the emotional feature as a second sample pair, and acquire an associated speech content feature corresponding to the speech content from the sample data, determine the emotional feature corresponding to the associated speech content feature and the speech content feature as a third sample pair;

[0023] The prediction module performs prediction processing on the first sample pair, the second sample pair and the third sample pair respectively to obtain the predicted mixed shape coefficients corresponding to the first sample pair, the predicted mixed shape coefficients corresponding to the second sample pair and the predicted mixed shape coefficients corresponding to the third sample pair.

[0024] In a second aspect, the present application also provides a method for voice-driven facial animation, the method comprising:

[0025] Acquire target audio data, input the target audio data into a mixed shape coefficient prediction network, and predict a mixed shape coefficient corresponding to the target audio data;

[0026] Using the mixed shape coefficient to drive a prefabricated virtual character facial model;

[0027] Wherein, the mixed shape coefficient prediction network is trained using any of the network training methods described above.

[0028] In one embodiment, the step of using the mixed shape coefficient to drive a prefabricated virtual character face model comprises:

[0029] The mixed shape coefficients are smoothed using a preset window size to obtain the smoothed mixed shape coefficients, and the smoothed mixed shape coefficients are used to drive the prefabricated virtual character facial model.

[0030] In a third aspect, the present application further provides a network training device, the device comprising:

[0031] An acquisition module, used for acquiring a target frame image and corresponding audio data from sample video data;

[0032] A mapping module, used to identify a face image from the target frame image, perform image processing on the face image through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and map the target facial motion vector into a target mixed shape coefficient using a target mapping matrix;

[0033] A prediction module, configured to input the audio data into an initial mixed shape coefficient prediction network to obtain a predicted mixed shape coefficient corresponding to the audio data;

[0034] A training module is used to train the initial mixed shape coefficient prediction network based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient to obtain a mixed shape coefficient prediction network.

[0035] In one embodiment, the initial mixed shape coefficient prediction network includes an audio feature extraction module and the prediction module, and the prediction module is specifically used to:

[0036] Obtaining the autocorrelation coefficient of the audio data using linear predictive coding;

[0037] Inputting the autocorrelation coefficient and the user-defined emotion intensity into the audio feature extraction module to obtain the emotion feature and speech content feature corresponding to the audio data;

[0038] The prediction module performs prediction processing on the emotion feature and the speech content feature to obtain the prediction mixed shape coefficient corresponding to the audio data.

[0039] In one embodiment, the device further comprises:

[0040] An extraction module, for extracting a 51-dimensional first blend shape morphological key set and a 53-dimensional target motion vector set using a neutral face model;

[0041] A reconstruction module, configured to reconstruct a first human face 3D model based on the 51-dimensional mixed shape morphological key set, and to reconstruct a second human face 3D model based on the 53-dimensional target motion vector set;

[0042] A training module is used to map the target position of the second three-dimensional face model to the first three-dimensional face model through an initial mapping matrix, and adjust the initial mapping matrix based on the difference between the mapped position and the target position of the first three-dimensional face model to obtain a target mapping matrix.

[0043] In one embodiment, the prediction module is specifically used for:

[0044] Taking the emotion feature and the speech content feature as a first sample pair;

[0045] Determining an emotion label corresponding to the emotion feature, and determining a speech content corresponding to the speech content feature;

[0046] Acquire an associated emotional feature corresponding to the emotional tag from the sample data, determine the speech content feature corresponding to the associated emotional feature and the emotional feature as a second sample pair, and acquire an associated speech content feature corresponding to the speech content from the sample data, determine the emotional feature corresponding to the associated speech content feature and the speech content feature as a third sample pair;

[0047] The prediction module performs prediction processing on the first sample pair, the second sample pair and the third sample pair respectively to obtain the predicted mixed shape coefficients corresponding to the first sample pair, the predicted mixed shape coefficients corresponding to the second sample pair and the predicted mixed shape coefficients corresponding to the third sample pair.

[0048] In a fourth aspect, the present application further provides a device for voice-driven facial animation, the device comprising:

[0049] A prediction module, used to obtain target audio data, input the target audio data into a mixed shape coefficient prediction network, and predict the mixed shape coefficient corresponding to the target audio data;

[0050] A driving module, used for driving a prefabricated virtual character face model using the mixed shape coefficient;

[0051] Wherein, the mixed shape coefficient prediction network is trained using any of the network training methods described above.

[0052] In one embodiment, the driving module is specifically used for:

[0053] The mixed shape coefficients are smoothed using a preset window size to obtain the smoothed mixed shape coefficients, and the smoothed mixed shape coefficients are used to drive the prefabricated virtual character facial model.

[0054] In a fifth aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above network training methods when executing the computer program.

[0055] In a sixth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods of voice-driven facial animation when executing the computer program.

[0056] In a seventh aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above network training methods when executed by a processor.

[0057] In an eighth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above methods of voice-driven facial animation.

[0058] In a ninth aspect, the present application also provides a computer program product, including a computer program, which implements any of the above network training methods when executed by a processor.

[0059] In a tenth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above methods of voice-driven facial animation.

[0060] The network training method and device, the method and device for voice-driven facial animation, the computer device, the computer-readable storage medium and the computer program product provided by the embodiments of the present disclosure are used to map the facial motion vector corresponding to the target frame image in the three-dimensional deformable model to the preset 51-dimensional facial mixed shape morphological key through the pre-trained target mapping matrix, and use this as the supervision information of the audio data corresponding to the target frame image to train the mixed shape coefficient prediction network, which is used to predict the mixed shape coefficient corresponding to the audio data, and the mixed shape coefficient is used to drive the prefabricated virtual character facial model. In this way, there is no need to collect a 3D facial animation data set, which can overcome the problem of high data set production cost in the fusion of 3D digital human expression drive, and realize the mapping of facial motion vectors and mixed shape coefficients through the preset target mapping matrix, which can overcome the problem that the randomness of emotional facial expressions is too large to fit and the problem that it is difficult to adapt to the role produced by the artist, and can reduce costs and improve the driving effect of voice-driven animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A schematic diagram of a flow chart of a network training method in one embodiment;

[0063] Figure 2 A schematic diagram of training an expression prediction network in one embodiment;

[0064] Figure 3 is a flow chart of step 106 in one embodiment;

[0065] Figure 4 is a flow chart of step 306 in one embodiment;

[0066] Figure 5 A schematic diagram of a flow chart of a network training method in one embodiment;

[0067] Figure 6 A flowchart of a method for voice-driven facial animation in one embodiment;

[0068] Figure 7 A schematic diagram of a method for voice-driven facial animation in one embodiment;

[0069] Figure 8 A schematic diagram of a method for voice-driven facial animation in one embodiment;

[0070] Fig. 9 is a structural block diagram of a network training device in one embodiment;

[0071] Fig.10 A structural block diagram of a device for voice-driven facial animation in one embodiment;

[0072] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] In one embodiment, Figure 1 As shown, a network training method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 102 to 108, wherein:

[0075] Step 102: Acquire a target frame image and corresponding audio data from sample video data.

[0076] In an embodiment of the present disclosure, a sample set may be pre-constructed, and the sample set includes a plurality of sample video data, wherein the sample video data is data including video and audio, wherein the video includes a face image. For any sample video data, a target frame image and corresponding audio data may be obtained from the sample video data. Exemplarily, the audio corresponding to the sample video data may be resampled to audio with a frequency of 16000 Hz. Since the target animation frame rate is 30, a waveform of 33ms may be used as a sliding step length, 520ms may be used as a sliding window size, and a window offset of 10ms may be randomly set to obtain a more generalized effect. Thus, a frame image is extracted from the sample video data as a target frame image based on a sliding window, and audio data is extracted as audio data corresponding to the target frame image.

[0077] Step 104, a face image is identified from the target frame image, the face image is processed through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and the target facial motion vector is mapped into a target mixed shape coefficient using a target mapping matrix.

[0078] In the disclosed embodiment, for any target frame image, a face region can be detected, and the face region can be cropped from the target frame image to obtain a face image. Exemplarily, the face region can be detected in advance using a face detection algorithm S³FD (Single Shot Scale-invariant Face Detector) on the extracted target frame image, extracting the face detection frame region in each frame image and scaling the face detection region to a cropped face image I_crop with a size of 256×256.

[0079] It should be noted that the face detection method is not specifically limited in the embodiments of the present disclosure. In fact, convolutional neural networks, YOLO (You Only Look Once), HOG (Histogram of Oriented Gradients), etc. can also be used to detect the face area from the target frame image. The technical personnel in this field can select the corresponding face detection method as needed.

[0080] After the face image is recognized, the face image can be input into a pre-trained expression prediction network for image processing to obtain the target facial motion vector corresponding to the face image in the three-dimensional deformable model. The expression prediction network is a pre-trained network for predicting facial motion vectors. Figure 2 As shown, the training process is as follows:

[0081] In the disclosed embodiment, the initial expression prediction network is a network with a ResNet50 architecture, which is a deep convolutional neural network used to extract features from input images. The input of the initial expression prediction network is a sample face image, and the output is a facial motion vector corresponding to the sample face image, which is used to describe expression-related changes in the three-dimensional model of the face.

[0082] During the training process, the sample face image can be input into ResNet50 to extract the face features, and the features can be mapped to the face motion vector through the fully connected layer, etc. After obtaining the face motion vector, the shape coefficient (describing the texture shape of the face), the illumination coefficient (describing the lighting conditions of the face), and the camera parameters (describing the position and viewing angle of the camera, that is, the posture of the face) of the sample face image can be extracted, and based on the face motion vector, the shape coefficient of the face image, the illumination coefficient and the camera parameters, the reconstructed face image can be generated through the differentiable rendering technology.

[0083] The initial expression prediction network can be trained by pixel-level loss, perceptual-level loss and facial key point loss, wherein the pixel-level loss may include: calculating the pixel difference between the sample face image and the reconstructed face image to measure the direct similarity of the images. The perceptual-level loss may include: using the difference between the sample face image extracted by the pre-trained deep network and the reconstructed face image, considering the semantic information of the image. The facial key point loss may include: calculating the position difference between the facial key points of the sample face image and the reconstructed face image to ensure that the reconstructed face is consistent with the sample face image in the key point position. In this way, the initial expression prediction network can be comprehensively optimized by combining the pixel-level loss, the perceptual-level loss and the facial key point loss, so that the facial image reconstructed based on the facial motion vector predicted by the initial expression prediction network is as close as possible to the original sample face image in terms of pixel level, semantic level and key point position, thereby improving the reconstruction accuracy of the facial image.

[0084] After extracting the target facial motion vector corresponding to the face image, the target facial motion vector can be mapped to the target mixed shape coefficient using a pre-trained target mapping matrix for mapping between the facial motion vector and the mixed shape coefficient, wherein the training process of the target mapping matrix will be described in detail in the following embodiments, and will not be repeated in the embodiments of the present disclosure. Among them, the mixed shape is a three-dimensional model used to describe the changes in facial expressions, each mixed shape corresponds to a specific expression (such as smiling, blinking, etc.), and the mixed shape coefficient is a weight parameter that describes these changes in expressions.

[0085] Step 106: Input the audio data into an initial mixed shape coefficient prediction network to obtain predicted mixed shape coefficients corresponding to the audio data.

[0086] Step 108 , based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient, an initial mixed shape coefficient prediction network is trained to obtain a mixed shape coefficient prediction network.

[0087] In the disclosed embodiment, audio data can be input into the initial mixed shape coefficient prediction network for prediction to obtain the predicted mixed shape coefficient corresponding to the audio data. The target mixed shape coefficient obtained by mapping the target facial motion vector corresponding to the face image is used as supervision information to determine the difference between the target mixed shape coefficient and the predicted mixed shape coefficient in each sample, and the initial mixed shape coefficient prediction network is trained based on the difference to obtain the mixed shape coefficient prediction network.

[0088] Exemplarily, a neural network model can be pre-set as an initial mixed shape coefficient prediction network, which is used to predict the mixed shape coefficient. At the beginning of training, this network is untrained and the parameters are randomly initialized. By minimizing the difference between the target mixed shape coefficient and the predicted mixed shape coefficient, the initial mixed shape coefficient prediction network is trained so that it can accurately predict the mixed shape coefficient. For example: the difference between the target mixed shape coefficient and the predicted mixed shape coefficient can be calculated using but not limited to the following loss functions: mean-square error (MSE), L1 loss function, Smooth L1 loss function, etc. The gradient of the loss function to the network parameters is calculated by the back propagation algorithm. Use optimization algorithms (such as Adam, SGD (Stochastic Gradient Descent, stochastic gradient descent), etc.) to update the network parameters to minimize the loss function value. After the training is completed, a mixed shape coefficient prediction network that can be used to predict the mixed shape coefficient of a new face image is obtained, thereby reconstructing facial expressions.

[0089] The network training method provided by the embodiment of the present disclosure is adopted, and the facial motion vector corresponding to the target frame image in the three-dimensional deformable model is mapped to the preset 51-dimensional facial mixed shape morphological key through the pre-trained target mapping matrix, and this is used as the supervision information of the audio data corresponding to the target frame image to train the mixed shape coefficient prediction network, and the mixed shape coefficient prediction network is used to predict the mixed shape coefficient corresponding to the audio data, and the mixed shape coefficient is used to drive the prefabricated virtual character facial model. In this way, there is no need to collect 3D facial animation data sets, and the problem of high data set production cost in 3D digital human expression drive fusion can be overcome. The mapping of facial motion vectors and mixed shape coefficients is realized through the preset target mapping matrix, which can overcome the problem that the randomness of emotional facial expressions is too large to fit and the problem that it is difficult to adapt to the characters produced by artists, and can reduce costs and improve the driving effect of voice-driven animation.

[0090] In an exemplary embodiment, the initial mixed shape coefficient prediction network includes an audio feature extraction module and a prediction module, referring to Figure 3 As shown, in step 106, the audio data is input into the initial mixed shape coefficient prediction network to obtain the predicted mixed shape coefficient corresponding to the audio data, which may include the following steps 302 to 306, wherein:

[0091] Step 302, using linear predictive coding to obtain the autocorrelation coefficient of the audio data;

[0092] Step 304, inputting the autocorrelation coefficient and the user-defined emotion intensity into an audio feature extraction module to obtain the emotion feature and speech content feature corresponding to the audio data;

[0093] Step 306: predict the emotion features and the speech content features through a prediction module to obtain predicted mixed shape coefficients corresponding to the audio data.

[0094] In the disclosed embodiment, linear prediction coding can be used to obtain the autocorrelation coefficient of audio data. Exemplarily, the autocorrelation coefficient size can be set to 64×32. The user can pre-set the intensity coefficient of the emotion (user-defined emotion intensity), for example, the smile coefficient can be taken from 0 to 1. After the autocorrelation coefficient of the audio data and the user-defined emotion intensity are input into the audio feature extraction module, the audio feature extraction module can be used to extract the emotion features and speech content features corresponding to the audio data, and further, the prediction module can be used to predict the emotion features and speech content features, and the prediction mixed shape coefficient corresponding to the audio data can be predicted.

[0095] Among them, the audio feature extraction module can adopt a network with a ResNet50 architecture, and the prediction module can adopt a variational encoder. In the disclosed embodiment, the emotional features and speech content features can be used as the mean and variance of the normal distribution in the variational encoder, respectively, to reconstruct the predicted mixed shape coefficients. In the decoding stage of the variational encoder, the input data can be reconstructed using a latent vector sampled from a normal distribution in the latent space. In this scenario, the sampled latent vector can be regarded as an instance of a mixed shape coefficient. By adjusting the mean and variance vectors, the characteristics and distribution of the reconstructed shape coefficients can be controlled. For example, by changing the mean vector, different central trends of the shape coefficients can be explored; and by adjusting the variance vector, the range and diversity of the shape coefficients can be controlled.

[0096] In the disclosed embodiment, the output features in the initial mixed shape coefficient prediction network can be decoupled into emotional features and speech content features through the audio feature extraction module, so as to achieve the effect of decoupling the speech emotion features from the speech content features, improve the prediction accuracy of the mixed shape coefficients, and thereby improve the animation driving effect.

[0097] In an exemplary embodiment, referring to Figure 4 As shown, in step 306, the prediction module performs prediction processing on the emotion feature and the speech content feature to obtain the prediction mixed shape coefficient corresponding to the audio data, which may include the following steps 402 to 408, wherein:

[0098] Step 402, taking the emotion feature and the speech content feature as a first sample pair;

[0099] Step 404, determining the emotion label corresponding to the emotion feature, and determining the speech content corresponding to the speech content feature;

[0100] Step 406, obtaining the associated emotional feature corresponding to the emotional label from the sample data, determining the speech content feature and the emotional feature corresponding to the associated emotional feature as a second sample pair, and obtaining the associated speech content feature corresponding to the speech content from the sample data, determining the emotional feature and the speech content feature corresponding to the associated speech content feature as a third sample pair;

[0101] Step 408, predict the first sample pair, the second sample pair and the third sample pair respectively through the prediction module to obtain the predicted mixed shape coefficients corresponding to the first sample pair, the predicted mixed shape coefficients corresponding to the second sample pair and the predicted mixed shape coefficients corresponding to the third sample pair.

[0102] In the embodiment of the present disclosure, during the training process, the speech content features and the emotion features can be decoupled by exchanging the emotion features with different emotion labels and the same speech content, and exchanging the speech content features with different speech content and the same emotion labels.

[0103] Exemplarily, in the process of exchanging emotional features and speech content features, there are two exchange methods: In method 1, emotional features with different emotional labels but corresponding to the same speech content are exchanged, with the purpose of allowing the network to learn the separation of emotional features and speech content features. By exchanging emotional features but keeping the speech content features unchanged, the model can be forced to learn the independence between emotion and content. In method 2, speech content features with different speech content but the same emotional labels can be exchanged. By exchanging speech content features but keeping the emotional labels unchanged, the network can further learn the independent representation between emotion and content.

[0104] Through the above exchange process, the network will gradually learn to process speech content features and emotional features separately. This means that given a piece of speech, the network can recognize its content and emotion separately, and the two are represented independently in the model. During the training process, the prediction module will continuously receive new training sample pairs obtained through exchange. These sample pairs help the prediction module gradually learn how to process content and emotional features independently. Supervisory information (mixed shape coefficients) plays a key role in this process. It ensures that the model can correctly recognize and process mixed states of different emotions and contents during training.

[0105] Exemplarily, the emotion feature and speech content feature corresponding to the current audio data may be used as sample pair 1, and the corresponding supervision information is the target mixed shape coefficient 1. Sample pair 2 and sample pair 3 may be obtained from the sample library, wherein the emotion feature in sample pair 2 corresponds to the same emotion label as the emotion feature in sample pair 1, and the speech content feature in sample pair 3 corresponds to the same speech content as the speech content feature in sample pair 1, wherein the supervision information of sample pair 2 is the target mixed shape coefficient 2, and the supervision information of sample pair 3 is the target mixed shape coefficient 3.

[0106] The speech content features in sample pair 1 and sample pair 2 can be interchanged to obtain sample pair 4 and sample pair 5, and the emotional features in sample pair 1 and sample pair 3 can be interchanged to obtain sample pair 5 and sample pair 7, wherein the supervision information of sample pair 4 and sample pair 6 is the target mixing shape coefficient 1, the supervision information of sample pair 5 is the target mixing shape coefficient 2, and the supervision information of sample pair 7 is the target mixing shape coefficient 3.

[0107] The prediction module predicts the emotional features and speech content features in each sample pair respectively to obtain the predicted mixed shape coefficient, and determines the loss of the prediction module based on the supervision information of each sample pair, and adjusts the network parameters of the prediction module and / or the audio feature extraction module based on the loss. After the training is completed, the mixed shape coefficient prediction network is obtained.

[0108] Exemplarily, for the audio data corresponding to the current face image, the emotional feature (hereinafter referred to as emotional feature 1 for clear description) and the speech content feature (hereinafter referred to as speech content feature 1 for clear description) obtained by feature extraction of the current audio data are used as the first sample pair, and the emotional label corresponding to the emotional feature 1 and the speech content corresponding to the speech content feature 1 are determined. It should be noted that the embodiment of the present application does not specifically limit the method of determining the emotional label corresponding to the emotional feature 1 and the speech content corresponding to the speech content feature 1, such as using a classifier, a pre-trained recognition network, etc.

[0109] Obtain from the sample library at least one associated emotional feature corresponding to the same emotional label as the current emotional feature 1 (hereinafter referred to as emotional feature 2 for clear description), and obtain from the sample library at least one associated speech content feature corresponding to the same speech content as the current speech content feature 1 (hereinafter referred to as speech content feature 2 for clear description), construct the emotional feature 2 and the speech content feature 1 into a second sample pair, and construct the emotional feature 1 and the speech content feature into a third sample pair.

[0110] The first sample pair, the second sample pair and the third sample pair are used as training data, and the target mixed shape coefficient determined based on the current face image is used as supervision information to train the prediction module and the audio feature extraction module. After the training is completed, a mixed shape coefficient prediction network can be obtained.

[0111] In an exemplary embodiment, referring to Figure 5 As shown, the method may further include the following steps 502 to 506, wherein:

[0112] Step 502, using the neutral face model, extracting a 51-dimensional first blend shape morphological key set and a 53-dimensional target motion vector set;

[0113] Step 504, reconstructing a first human face 3D model based on the 51-dimensional mixed shape morphological key set, and reconstructing a second human face 3D model based on the 53-dimensional target motion vector set;

[0114] Step 506, maps the target position of the second face three-dimensional model to the first face three-dimensional model through an initial mapping matrix to obtain a mapping mixed shape morphological coefficient, and adjusts the initial mapping matrix according to the difference between the mapping mixed shape morphological coefficient and the mixed shape morphological coefficient corresponding to the target position in the first face three-dimensional model to obtain a target mapping matrix.

[0115] In the disclosed embodiment, a neutral face model of a three-dimensional deformable model is used as a base, and a 51-dimensional first blended shape morphological key set and a 53-dimensional target motion vector set (50 dimensions represent facial expression coefficients and 3 dimensions represent jaw posture coefficients) are produced according to the principle of FACS (Facial Action Coding System). Among them, the neutral face model is a face model with a neutral expression. Both the 51-dimensional blended shape morphological key and the 53-dimensional facial motion vector involve the description of shape and motion, but the blended shape morphological key focuses more on shape blending in three-dimensional modeling and animation, while the facial motion vector focuses more on the analysis and recognition of facial expressions.

[0116] A 51-dimensional set of mixed shape morphological keys can be pre-made, and these morphological keys describe different facial expressions (such as smiling, frowning, etc.), and a 53-dimensional set of target motion vectors can be made, and these vectors describe subtle facial motion changes (such as blinking, head rotation, etc.). Among them, the 51-dimensional set of mixed shape morphological keys and the 53-dimensional set of target motion vectors can be obtained by solving equations based on a neutral face model (for example, by solving in a nonlinear optimization manner), and the equation solving process is not elaborated in the embodiments of the present disclosure.

[0117] After obtaining a 51-dimensional mixed shape morphological key set and a 53-dimensional target motion vector set, the 51-dimensional mixed shape morphological key can be applied to a neutral face model to reconstruct a three-dimensional model with corresponding expression changes as the first three-dimensional face model, and the 53-dimensional target motion vector set can be applied to the neutral face model to reconstruct a three-dimensional model with corresponding motion changes as the second three-dimensional face model.

[0118] The target position is a pre-set position, which is the vertex position of the first face 3D model and the second face 3D model. The vertex position of the second face 3D model can be obtained, and the vertex position can be mapped to the first face 3D model through the initial mapping matrix to obtain the corresponding mapping mixed shape morphological coefficient, and the mixed shape morphological coefficient corresponding to the vertex position in the first face 3D model is obtained, and the difference between the two (such as the mean square error) is determined, and based on the difference between the two, the initial mapping matrix is ​​adjusted using an optimization algorithm (such as gradient descent, least squares method, etc.) to better map the facial motion vector in the second face 3D model to the mixed shape morphological coefficient in the first face 3D model.

[0119] The network training method provided in the embodiment of the present disclosure can map the expression parameters (facial motion vectors) to a 51-dimensional mixed shape morphological key with practical significance through a training target mapping matrix, thereby solving the problem of using a three-dimensional deformable model to produce too high a data set, that is, the three-dimensional deformable model cannot be adapted to the mixed shape morphological key produced by an artist because the expression and shape parameters have no actual physical meaning, thereby improving the animation driving effect.

[0120] In one embodiment, Figure 6 As shown, a method for voice-driven facial animation is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 602 to 604, wherein:

[0121] Step 602, obtaining target audio data, inputting the target audio data into a mixed shape coefficient prediction network, and predicting the mixed shape coefficients corresponding to the target audio data;

[0122] Step 604, using the mixed shape coefficient to drive the prefabricated virtual character face model;

[0123] The mixed shape coefficient prediction network is trained using any of the network training methods in the aforementioned embodiments.

[0124] In the disclosed embodiment, after obtaining the target audio data, the target audio data can be input into the mixed shape coefficient prediction network to predict the mixed shape coefficient corresponding to the target audio data, and the mixed shape coefficient is used to drive the prefabricated virtual character face model. Since the mixed shape coefficient prediction network in the disclosed embodiment maps the expression and shape parameters that have no actual physical meaning in the three-dimensional deformable model to the adapted mixed shape morphological key through the target mapping matrix during the training process, the problem that the three-dimensional deformable model cannot adapt to the character produced by the artist is solved, and animation driving is realized through the three-dimensional deformable model, thereby reducing costs and improving the driving effect of voice-driven animation.

[0125] In an exemplary embodiment, using a mixed shape coefficient to drive a prefabricated virtual character facial model may include: smoothing the mixed shape coefficient using a preset window size to obtain a smoothed mixed shape coefficient, and using the smoothed mixed shape coefficient to drive the prefabricated virtual character facial model.

[0126] In the disclosed embodiment, after the mixed shape coefficients are inferred, a median filter with a preset window size (for example, the window size is set to 5) can be used to smooth the mixed shape coefficients in the time dimension, and the smoothed mixed shape coefficients can be used to drive the prefabricated virtual character facial model.

[0127] In order to enable those skilled in the art to better understand the embodiments of the present disclosure, the embodiments of the present disclosure are described below through specific examples.

[0128] The disclosed embodiment maps the expression coefficients of a three-dimensional deformable model to a hybrid shape morphological key with practical significance through affine deformation guided by local area key points. This method transforms the originally uncontrollable three-dimensional deformable model into a controllable hybrid shape morphological key, which is more conducive to effective manual intervention in facial animation. At the same time, this method can effectively adapt to the facial morphological keys produced by art. Compared with the method in the related art that directly uses speech frames and morphological key coefficients as the input and output of the deep network respectively, the disclosed embodiment effectively decouples emotions from content by exchanging and decoupling emotions and content during the training process, thereby solving the problem of the inadaptability of emotional facial animation inferred from speech, so as to achieve a more accurate and natural prediction effect. Reference Figure 7 and Figure 8 As shown, the method provided by the embodiment of the present disclosure may also include:

[0129] Mapping a 3D Morphable Model to a Blend Shape Shape Key. Neutral Face Model using a 3D Morphable Model As a base, a 51-dimensional blended shape bond set was created according to the principles of FACS. . Through the 53-dimensional facial motion vector (50 dimensions represent facial expression coefficients and 3 dimensions represent jaw posture coefficients) can reconstruct a 3D facial expression model , reconstruct the 3D model through the 51-dimensional shape key . use and The mean square error loss of the vertex positions between is trained with a shape of Mapping Matrix ,The facial motion vector can be mapped to the blending shape coefficient through the mapping matrix.

[0130] Detect the face area. Use the face detection algorithm S³FD (Single Shot Scale-invariant Face Detector) to detect the face area of ​​the original video to be inferred in advance, extract the face detection frame area in each frame and scale the face detection area to a size of Cropped face image .

[0131] The face image will be cropped As input, the facial motion vector in the 3D deformable model coefficients is obtained by predicting the network coefficients with the expression coefficients of the ResNet50 architecture. During the training process, the facial motion vector , the shape coefficient of the network estimate , illumination coefficient And camera parameters , the reconstructed face image can be obtained through differentiable rendering . Using cropped face images Reconstructing face images The three loss functions of pixel level loss, perception level loss and face key point loss are used to improve the accuracy of face reconstruction.

[0132] Audio preprocessing. The input audio is resampled to a frequency of 16000 Hz. Since the target animation frame rate is 30, a 33ms waveform is used as the sliding step length, 520ms as the sliding window size, and a 10ms window offset is randomly set to obtain a more generalized effect. Linear predictive coding is used to obtain the autocorrelation coefficient of each sliding window. The size of the autocorrelation coefficient of each window is .

[0133] Use the face image frames corresponding to each window as input to estimate With the mapping matrix Get the corresponding blend shape coefficients The audio window autocorrelation coefficient and the user-defined emotion intensity are used to As input, ResNet50 is used as a skeleton model to decouple the skeleton model output features into emotional features. Features of speech content , and respectively and The mixture shape coefficients are reconstructed as the mean and variance of the normal distribution in the variational encoder , the blend shape coefficient As supervision information, the mixed shape coefficients are exchanged during training. In the case of , exchange speech content features with different speech content and the same emotion label To achieve the decoupling of speech content and emotional features.

[0134] After the above process, the mixed shape coefficient is obtained. Finally, a median filter with a window of 5 is used to smooth the mixed shape coefficients in the temporal dimension, and the smoothed mixed shape coefficients are used to drive the prefabricated virtual character face model.

[0135] The method of voice-driven facial animation provided in the disclosed embodiment uses a three-dimensional deformable model to solve the problem of too high a dataset production result. However, the three-dimensional deformable model cannot be adapted to the mixed shape morphological key produced by the artist because the expression and shape parameters have no actual physical meaning. Therefore, in the disclosed embodiment, a fully connected layer is used as a model, and the model vertex coordinates are used as guiding information to map the expression parameters to a 52-dimensional expression mixed shape morphological key with practical significance. Emotional facial expressions are highly random, and the same voice may correspond to different facial expressions. In the disclosed embodiment, the audio-based facial expression estimation is trained by cross-exchanging and reconstructing different voice emotions and voice content during the training process to achieve the effect of decoupling voice emotion features from voice content features.

[0136] To sum up, compared with the methods in the prior art, the network training method provided by the embodiments of the present invention uses 2D images and facial key points as supervisory information, utilizes the differentiable rendering of the three-dimensional deformable model as supervisory information to fit the facial shape, and maps the expression parameters of the 3D deformable model to the preset 52-dimensional facial blended shape morphological keys. While improving the reasoning accuracy and the naturalness of expression, it does not require a large amount of manually labeled data, and provides a more reliable and effective solution for the voice-driven effect of 3D digital human emotional facial animation. It also utilizes a deep learning network to decouple the voice content and emotions and estimate the blended shape weights of the corresponding voice for driving facial animation.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a network training device for implementing the network training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more network training device embodiments provided below can refer to the limitations of the network training method above, and will not be repeated here.

[0139] In an exemplary embodiment, Fig. 9As shown, a network training device is provided, including: an acquisition module 902, a mapping module 904, a prediction module 906 and a training module 908, wherein:

[0140] An acquisition module 902 is used to acquire a target frame image and corresponding audio data from sample video data;

[0141] A mapping module 904 is used to identify a face image from the target frame image, perform image processing on the face image through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and map the target facial motion vector into a target mixed shape coefficient using a target mapping matrix;

[0142] A prediction module 906, configured to input the audio data into an initial mixed shape coefficient prediction network to obtain a predicted mixed shape coefficient corresponding to the audio data;

[0143] The training module 908 is used to train the initial mixed shape coefficient prediction network based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient to obtain a mixed shape coefficient prediction network.

[0144] The network training device provided by the embodiment of the present disclosure is used to map the facial motion vector corresponding to the target frame image in the three-dimensional deformable model to the preset 51-dimensional facial blended shape morphological key through the pre-trained target mapping matrix, and use this as the supervision information of the audio data corresponding to the target frame image to train the blended shape coefficient prediction network, which is used to predict the blended shape coefficient corresponding to the audio data, and the blended shape coefficient is used to drive the prefabricated virtual character facial model. In this way, there is no need to collect 3D facial animation data sets, which can overcome the problem of high data set production cost in 3D digital human expression drive fusion. The mapping of facial motion vectors and blended shape coefficients is achieved through the preset target mapping matrix, which can overcome the problem of excessive randomness of emotional facial expressions that are difficult to fit and difficult to adapt to the characters produced by artists, and can reduce costs and improve the driving effect of voice-driven animation.

[0145] In one embodiment, the initial mixed shape coefficient prediction network includes an audio feature extraction module and the prediction module, and the prediction module 906 is specifically used to:

[0146] Obtaining the autocorrelation coefficient of the audio data using linear predictive coding;

[0147] Inputting the autocorrelation coefficient and the user-defined emotion intensity into the audio feature extraction module to obtain the emotion feature and speech content feature corresponding to the audio data;

[0148] The prediction module performs prediction processing on the emotion feature and the speech content feature to obtain the prediction mixed shape coefficient corresponding to the audio data.

[0149] In one embodiment, the device further comprises:

[0150] An extraction module, for extracting a 51-dimensional first blend shape morphological key set and a 53-dimensional target motion vector set using a neutral face model;

[0151] A reconstruction module, configured to reconstruct a first human face 3D model based on the 51-dimensional mixed shape morphological key set, and to reconstruct a second human face 3D model based on the 53-dimensional target motion vector set;

[0152] A training module is used to map the target position of the second three-dimensional face model to the first three-dimensional face model through an initial mapping matrix, and adjust the initial mapping matrix based on the difference between the mapped position and the target position of the first three-dimensional face model to obtain a target mapping matrix.

[0153] In one embodiment, the prediction module 906 is specifically used for:

[0154] Taking the emotion feature and the speech content feature as a first sample pair;

[0155] Determining an emotion label corresponding to the emotion feature, and determining a speech content corresponding to the speech content feature;

[0156] Acquire an associated emotional feature corresponding to the emotional tag from the sample data, determine the speech content feature corresponding to the associated emotional feature and the emotional feature as a second sample pair, and acquire an associated speech content feature corresponding to the speech content from the sample data, determine the emotional feature corresponding to the associated speech content feature and the speech content feature as a third sample pair;

[0157] The prediction module performs prediction processing on the first sample pair, the second sample pair and the third sample pair respectively to obtain the predicted mixed shape coefficients corresponding to the first sample pair, the predicted mixed shape coefficients corresponding to the second sample pair and the predicted mixed shape coefficients corresponding to the third sample pair.

[0158] Based on the same inventive concept, the embodiment of the present application also provides a voice-driven facial animation device for implementing the voice-driven facial animation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more voice-driven facial animation device embodiments provided below can refer to the limitations of the voice-driven facial animation method above, and will not be repeated here.

[0159] In an exemplary embodiment, Fig.10 As shown, a device for voice-driven facial animation is provided, comprising: a prediction module 1002 and a driving module 1004, wherein:

[0160] A prediction module 1002 is used to obtain target audio data, input the target audio data into a mixed shape coefficient prediction network, and predict the mixed shape coefficient corresponding to the target audio data;

[0161] A driving module 1004, configured to drive a prefabricated virtual character face model using the mixed shape coefficient;

[0162] The mixed shape coefficient prediction network is trained using the network training method described in any one of the aforementioned embodiments.

[0163] In one embodiment, the driving module 1004 is specifically used for:

[0164] The mixed shape coefficients are smoothed using a preset window size to obtain the smoothed mixed shape coefficients, and the smoothed mixed shape coefficients are used to drive the prefabricated virtual character facial model.

[0165] Each module in the above network training device and voice-driven facial animation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0166] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a network training method or a voice-driven facial animation method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0167] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0169] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0170] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0172] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0174] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A network training method, characterized in that: The method comprises: Obtaining a target frame image and corresponding audio data from sample video data; Recognize a face image from the target frame image, perform image processing on the face image through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and map the target facial motion vector into a target mixed shape coefficient using a target mapping matrix; Inputting the audio data into an initial mixed shape coefficient prediction network to obtain predicted mixed shape coefficients corresponding to the audio data; Based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient, the initial mixed shape coefficient prediction network is trained to obtain a mixed shape coefficient prediction network.

2. The method according to claim 1, characterized in that The initial mixed shape coefficient prediction network includes an audio feature extraction module and a prediction module, and the step of inputting the audio data into the initial mixed shape coefficient prediction network to obtain the predicted mixed shape coefficient corresponding to the audio data includes: Obtaining the autocorrelation coefficient of the audio data using linear predictive coding; Inputting the autocorrelation coefficient and the user-defined emotion intensity into the audio feature extraction module to obtain the emotion feature and speech content feature corresponding to the audio data; The prediction module performs prediction processing on the emotion feature and the speech content feature to obtain the prediction mixed shape coefficient corresponding to the audio data.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Using the neutral face model, extract a 51-dimensional blend shape morphological key set and a 53-dimensional target motion vector set; Reconstructing a first 3D face model based on the 51-dimensional mixed shape morphological key set, and reconstructing a second 3D face model based on the 53-dimensional target motion vector set; The target position of the second three-dimensional face model is mapped to the first three-dimensional face model through an initial mapping matrix to obtain a mapping mixed shape morphological coefficient, and according to the difference between the mapping mixed shape morphological coefficient and the mixed shape morphological coefficient corresponding to the target position in the first three-dimensional face model, the initial mapping matrix is ​​adjusted to obtain a target mapping matrix.

4. The method according to claim 2, characterized in that: The predicting module predicts the emotion feature and the speech content feature to obtain a predicted mixed shape coefficient corresponding to the audio data, including: Taking the emotion feature and the speech content feature as a first sample pair; Determining an emotion label corresponding to the emotion feature, and determining a speech content corresponding to the speech content feature; Acquire an associated emotional feature corresponding to the emotional tag from the sample data, determine the speech content feature corresponding to the associated emotional feature and the emotional feature as a second sample pair, and acquire an associated speech content feature corresponding to the speech content from the sample data, determine the emotional feature corresponding to the associated speech content feature and the speech content feature as a third sample pair; The prediction module performs prediction processing on the first sample pair, the second sample pair and the third sample pair respectively to obtain the predicted mixed shape coefficients corresponding to the first sample pair, the predicted mixed shape coefficients corresponding to the second sample pair and the predicted mixed shape coefficients corresponding to the third sample pair.

5. A method for voice-driven facial animation, characterized in that: The method comprises: Acquire target audio data, input the target audio data into a mixed shape coefficient prediction network, and predict a mixed shape coefficient corresponding to the target audio data; Using the mixed shape coefficient to drive a prefabricated virtual character facial model; Wherein, the mixed shape coefficient prediction network is trained using the network training method described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that The method of using the mixed shape coefficient to drive the prefabricated virtual character face model comprises: The mixed shape coefficients are smoothed using a preset window size to obtain the smoothed mixed shape coefficients, and the smoothed mixed shape coefficients are used to drive the prefabricated virtual character facial model.

7. A network training device, characterized in that: The device comprises: An acquisition module, used to acquire a target frame image and corresponding audio data from sample video data; A mapping module, used to identify a face image from the target frame image, perform image processing on the face image through an expression prediction network to obtain a target facial motion vector corresponding to the face image, and map the target facial motion vector into a target mixed shape coefficient using a target mapping matrix; A prediction module, configured to input the audio data into an initial mixed shape coefficient prediction network to obtain a predicted mixed shape coefficient corresponding to the audio data; A training module is used to train the initial mixed shape coefficient prediction network based on the difference between the target mixed shape coefficient and the predicted mixed shape coefficient to obtain a mixed shape coefficient prediction network.

8. A device for voice-driven facial animation, characterized in that: The device comprises: A prediction module, used to obtain target audio data, input the target audio data into a mixed shape coefficient prediction network, and predict the mixed shape coefficient corresponding to the target audio data; A driving module, used for driving a prefabricated virtual character face model using the mixed shape coefficient; Wherein, the mixed shape coefficient prediction network is trained using the network training method described in any one of claims 1 to 4.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 or 5 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 or 5 to 6 are implemented.

Images