Method and system for separating emotions and driving facial animation synthesis based on audio features
By extracting and separating emotion and identity features from speech audio and combining them with reference facial images to generate facial animation, the problem of insufficient emotion feature separation in existing technologies is solved, and more realistic facial animation synthesis is achieved.
Patent Information
- Application Number
- CN202310401054.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-14
- Publication Date
- 2025-09-19
- Estimated Expiration
- 2043-04-14
AI Technical Summary
Existing audio-driven facial animation synthesis methods have difficulty in effectively separating emotional features from conversation audio, resulting in deficiencies in the authenticity and accuracy of the generated videos.
By extracting audio features from speech audio and combining them with reference facial images to separate content information, identity information and emotional information, a synthetic facial image is generated using a neural network. The iterative nearest point algorithm and orthogonal projection technology are used to process facial landmarks. The long short-term memory network and self-attention encoder are used to capture temporal information. The emotional and identity features are separated through contrastive learning. Finally, a synthetic facial image is generated using a generative network.
It significantly improves the realism and accuracy of facial animation synthesis, enables more fine-grained control of facial expressions, and generates more realistic videos.
Smart Images
Figure CN116403144B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and image synthesis, and in particular to a method and system for separating emotions based on audio features and driving facial animation synthesis. Background Art
[0002] Audio-driven facial animation synthesis is a hot problem in the field of computer vision. This problem can be described as, given a reference face image and an audio clip of a person talking, generating an animation of the face in the reference image corresponding to the audio clip while talking.
[0003] In recent years, a variety of audio-driven facial animation synthesis methods have emerged. These methods primarily utilize neural networks to extract features from audio, then convert the audio features into an intermediate representation. This intermediate representation is then superimposed with a reference image in some manner, and the resulting video is generated through a generative network. The resulting video has achieved significant improvements in realism and accuracy. However, in actual conversations, due to the influence of emotion, even though the content of the conversation is the same, the audio features such as pitch are not exactly the same. Therefore, how to separate the emotional component of the audio from the conversation audio and control the facial expressions of the generated video based on this separated emotional component to synthesize more realistic and accurate facial animation videos is an urgent problem in the industry. Summary of the Invention
[0004] In order to solve at least one of the above technical problems, the present invention proposes a method for separating emotions based on audio features and driving facial animation synthesis.
[0005] The purpose of the present invention is achieved through the following technical solutions:
[0006] The present invention provides a method for separating emotions based on audio features and driving facial animation synthesis, comprising the following steps:
[0007] S1. Given a speech audio segment to be animated and a dynamic reference facial image;
[0008] S2, extracting audio features from the speech audio and facial landmarks from the reference facial image;
[0009] S3, separating content information, identity information and emotion information from the extracted audio features combined with the reference facial image;
[0010] S4, obtaining the coordinates of the landmark points of the predicted image by combining the content information, identity information, emotion information and facial landmark points of the audio features through a neural network;
[0011] S5. Constructing a landmark point map according to the landmark point coordinates of the predicted image;
[0012] S6. Superimpose the landmark point map with the image in the reference facial image, and use the generative network to generate a synthesized facial image.
[0013] As a further improvement, in step S2, audio features are extracted from the speech audio. First, the speech audio is preprocessed, and the preprocessed speech audio is extracted to obtain the number of audio frames of the audio features and the dimension of each frame of the audio features.
[0014] As a further improvement, in step S2, extracting facial landmarks from the reference facial image includes the following steps:
[0015] S21, extracting all three-dimensional facial landmark coordinates from the reference facial image;
[0016] S22, calculating the average coordinates of all three-dimensional facial landmark points;
[0017] S23, aligning the coordinates of each 3D facial landmark point with the average coordinate value using an iterative closest point algorithm;
[0018] S24. Project the three-dimensional facial landmark points onto the image space through orthogonal projection.
[0019] As a further improvement, in step S3, the separation of content information, identity information, and emotion information from the extracted audio features in combination with the reference facial image includes the following steps:
[0020] S31. Encode audio features using neural networks;
[0021] S32. Using a long short-term memory network to capture the temporal information of the audio features, obtain content information including content feature embedding;
[0022] S33, use two neural networks to convert audio features into emotion feature embedding and identity feature embedding respectively;
[0023] S34, training a long short-term memory network and a neural network in combination with the reference facial image by contrastive learning to separate the emotion information embedded with the emotion feature and the identity information embedded with the identity feature;
[0024] S35. Input the emotion information containing the emotion feature embedding and the identity information containing the identity feature embedding into the self-attention encoder to obtain the style information containing the style feature embedding.
[0025] As a further improvement, in step S34, the emotional information embedded with the emotional features and the identity information embedded with the identity features are separated by contrastive learning in combination with the reference facial image, including the following steps:
[0026] S341, selecting a first video segment and a second video segment from the reference facial image, and then selecting a video segment of the same speaker as in the reference facial image as a third video segment;
[0027] S342, performing comparative learning on the three video clips in a comparative learning model, and separating the emotional information including the emotional feature embedding according to the emotional feature embedding of the first video clip and the emotional feature embedding of the second video clip;
[0028] S343, predicting the softmax probability of expressions in the three video clips by the expression prediction network;
[0029] S344 , separating identity information including identity feature embedding according to the identity feature embedding of the first video segment and the identity feature embedding of the third video segment.
[0030] As a further improvement, in step S342, the sentiment loss formula of contrastive learning is as follows:
[0031]
[0032] in, represents the sentiment loss function, λ r，s is the cosine similarity between the softmax probabilities of the predicted expressions of the first and second video clips, d r，s represents the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the second video clip, λ r，p The cosine similarity between the softmax probabilities of the predicted expressions of the first and third video clips, d r，p is the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the third video clip.
[0033] As a further improvement, in step S343, the identity information including the identity feature embedding is separated based on the identity feature embedding of the first video clip and the identity feature embedding of the third video clip. The identity loss formula of the comparative learning is as follows:
[0034]
[0035] in, represents the identity loss function, represents the identity feature embedding of the first video clip, Represents the identity feature embedding of the third video clip.
[0036] As a further improvement, in step S4, the coordinates of the landmark points of the predicted image are obtained by combining the content information, identity information, emotion information and facial landmark points of the audio features through a neural network, which specifically includes the following steps:
[0037] S41, inputting the content information including the embedded content features into the neural network to obtain the content landmark point coordinate offset;
[0038] S42, inputting the emotional information including the emotional feature embedding and the style information including the style feature embedding into the neural network to obtain the coordinate offset of the style landmark point;
[0039] S43 , adding the content landmark point coordinate offset, the style landmark point coordinate offset, and the facial landmark points extracted from the reference facial image to obtain the landmark point coordinates of the predicted image.
[0040] As a further improvement, in step S6, after the synthetic facial image is generated by the generative network, the difference in pixel values between the synthetic facial image and the real facial image is compared, and the image loss function is as follows:
[0041]
[0042] in, represents the image loss function, I s represents the real face image, I t Represents a synthesized face image.
[0043] The present invention provides a method for separating emotions based on audio features and driving facial animation synthesis, comprising the following steps: S1, given a speech audio segment to be animated and a dynamic reference facial image; S2, extracting audio features from the speech audio and extracting facial landmarks from the reference facial image; S3, separating content information, identity information, and emotion information from the extracted audio features in combination with the reference facial image; S4, obtaining the landmark coordinates of the predicted image by combining the content information, identity information, emotion information, and facial landmarks of the audio features through a neural network; S5, constructing a landmark map based on the landmark coordinates of the predicted image; S6, superimposing the landmark map with the image in the reference facial image, and generating a synthesized facial image using a generative network. The present invention explicitly separates the speech content, emotion part, and speaker identity characteristics of the audio features in the speech audio, explicitly controls the facial expression of the reference facial image based on the separated emotion part features, and synthesizes realistic and accurate facial animation videos in a more fine-grained manner.
[0044] The present invention also provides a facial animation synthesis system, including any improvement of the above-mentioned method of separating emotions based on audio features and driving facial animation synthesis. Since the above-mentioned technical content is adopted, it should have the same or corresponding technical effects, so it will not be described in detail. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a schematic diagram of the process of the present invention;
[0046] Figure 2 It is a schematic diagram of the architecture of the present invention;
[0047] Figure 3A This is a schematic diagram of the first qualitative experiment of an embodiment of the present invention;
[0048] Figure 3B Schematic diagram of the second qualitative experiment of the embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other unless there is a conflict.
[0050] Combine Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for separating emotions based on audio features and driving facial animation synthesis, including the following steps:
[0051] S1. Given a speech audio segment to be animated and a dynamic reference facial image;
[0052] S2. Extract audio features from speech audio. First, preprocess the speech audio and extract the preprocessed speech audio to obtain audio features. Where T is the number of audio frames, the audio frames are divided into intervals of 20ms and a step size of 20ms, and D is the dimension of the audio features of each frame.
[0053] Pre-processing of speech audio includes pre-emphasis, framing, and windowing. Pre-emphasis: Because speech audio power will be greatly attenuated when it is affected by glottal excitation and oral and nasal radiation, and the energy corresponding to high frequencies in the speech audio spectrum is small, and the energy corresponding to low frequencies is large. The high-frequency portion of unprocessed speech audio is even more difficult to obtain. The purpose of pre-emphasis is to enhance the high-frequency portion, making the speech audio spectrum flat, and then the spectrum can be obtained with the same signal-to-noise ratio within the entire frequency band, which facilitates spectrum analysis. Pre-emphasis is optional but not limited to the use of a first-order high-pass filter. Framing: Speech signals are generally non-stationary. The purpose of framing is to make the signal stable within the frame for subsequent processing. During framing, in order to ensure a smooth transition between frames, there will be overlapping parts between the previous and next frames. Windowing: In speech signal processing, the windows that can be selected but are not limited to rectangular windows and Hamming windows are optional but not limited to them.
[0054] Extracting facial landmarks from a reference facial image. Since the head pose in a reference facial image is usually dynamic, to prevent changes in head pose from affecting the positions of facial landmarks, and to facilitate learning a stable mapping between audio features and facial landmarks, the influence of head pose is eliminated by aligning the face in the reference facial image to a consistent head pose. This involves the following steps:
[0055] S21. Extract all three-dimensional facial landmark coordinates from the reference facial image
[0056] S22, calculating the average coordinates of all three-dimensional facial landmark points;
[0057] S23, using an iterative closest point (ICP) algorithm to align the coordinates of each 3D facial landmark point with the average coordinate value, using an iterative optimization approach to select matching points based on spatial distance, and continuously adjusting the position of each 3D facial landmark point coordinate to minimize the cumulative distance from the average coordinate value;
[0058] S24. Project the three-dimensional facial landmark points onto the image space through orthogonal projection.
[0059] S3, separating content information, identity information, and emotion information from the extracted audio features in combination with the reference facial image, including the following steps:
[0060] S31. Use a neural network (Multilayer Perceptron, MLP) to encode audio features. The MLP fully connected neural network is a forward-structured neural network that maps a set of input vectors to a set of output vectors.
[0061] S32. Use the Long Short-Term Memory (LSTM) network to capture the audio feature timing information extracted from the speech audio and obtain the content feature embedding Feature embedding converts data into fixed-size feature representations for easier processing and computation. The LSTM (Long Short-Term Memory) network is a type of neural network, a time-recurrent neural network used to process and predict important events with very long intervals and delays in speech and audio time series. The formula involved in content feature embedding is as follows:
[0062] E content =LSTM(MLP c (F a ))
[0063] Among them, E content Representing the content feature embedding of speech audio, MLPc represents the content encoder used to obtain content feature embedding, F a Represents the audio features of speech audio.
[0064] S33. Use two neural networks to convert audio features into emotional feature embeddings and identity feature embedding The formulas involved in sentiment feature embedding and identity feature embedding are as follows:
[0065] E emotion =MLP e (F a )
[0066] E identity =MLP i (F a )
[0067] Among them, E emotion Represents the emotional feature embedding of speech audio, E identity Representing identity feature embeddings for speech audio, MLP e Represents the style encoder used to obtain emotional feature embedding, MLP i represents the style encoder used to obtain identity feature embedding.
[0068] S34. Training a long short-term memory network and a neural network by contrastive learning in combination with a reference facial image to separate the emotional information embedded with the emotional features and the identity information embedded with the identity features, including the following steps:
[0069] S341, select the first video segment V from the reference facial image r and the second video clip V s , and then select a video clip of the same speaker as the reference facial image as the third video clip V p , the third video clip V p It is optional but not limited to selecting from reference facial images, you can also select from other video clips of the same speaker, using V r and V p Separate identity feature embedding;
[0070] S342, the three video clips are contrasted in the contrastive learning model. In order to separate the emotional feature embedding, we use V r 、V s and V p Separate the emotional feature embedding and construct the emotional contrast loss. Assuming that the emotional states in different video clips of the same video are more similar than those in different videos, V r and V s Emotional feature embedding and Compared with V r and V p Emotional feature embedding and There is a greater similarity, according to the first video segment V r The emotional feature embedding of the second video clip V s The sentiment feature embedding is used to separate the sentiment information contained in the sentiment feature embedding.
[0071] S343. In order to add appropriate weights to the emotion contrast loss, the pre-trained expression prediction network predicts the softmax probability of the expressions in the three video clips and records it as e r 、e s and e p , the Softmax function is the last layer of the neural network, which accepts the input value from the previous layer of the network and then converts it into probability. The sentiment loss function is as follows:
[0072]
[0073] in, represents the sentiment loss function, λ r，s is the cosine similarity between the softmax probabilities of the predicted expressions of the first and second video clips, a r，s represents the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the second video clip, λ r，p The cosine similarity between the softmax probabilities of the predicted expressions of the first and third video clips, a r，p is the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the third video clip.
[0074] S344, separating the identity information including the identity feature embedding according to the identity feature embedding of the first video segment and the identity feature embedding of the third video segment, so that the first video segment V r The identity feature embedding and the third video segment V p The identity feature embedding is as similar as possible, and the identity loss function formula is as follows:
[0075]
[0076] in, represents the identity loss function, Represents the identity feature embedding of the first video clip. Represents the identity feature embedding of the third video clip.
[0077] S35, input the emotional information containing the emotional feature embedding and the identity information containing the identity feature embedding input into the self-attention encoder to obtain the style feature embedding Because emotional information and identity information have a long time dependency, the self-attention encoder is designed to capture the dependency between emotional information embedded in emotional features and identity information embedded in identity features. Speech audio involves the speaker's instantaneous emotions and the speaker's own identity sound features. These features affect the conversation style and thus the speaker's facial expressions. Therefore, a style encoder is used to separate the conversation style from the speech audio. The style encoder includes an MLP for emotional feature embedding. e , MLP for identity feature embedding i and MLP for style feature embedding s The formula involved in style feature embedding is as follows:
[0078] E style =Attn(E emotion , E identity )
[0079] Among them, E style Represents the style feature embedding of speech audio, and Attn represents the merging function of style feature embedding obtained by self-attention encoder.
[0080] S4, using facial landmarks as an intermediate representation from the speech audio to the reference facial image, and obtaining landmark coordinates of the predicted image by combining the content information, identity information, emotion information, and facial landmarks of the speech audio through a neural network, specifically comprising the following steps:
[0081] S41, inputting the content information containing the embedded content features into the neural network to obtain the content landmark point coordinate offset The formula involved is as follows:
[0082] D content =MLP t (E content )
[0083] Among them, D content Indicates the content marker coordinate offset, MLP t Represents the content encoder used to obtain the content marker point coordinate offset.
[0084] S42, input the emotional information containing the emotional feature embedding and the style information containing the style feature embedding into the neural network to obtain the coordinate offset of the style landmark point The formula involved is as follows:
[0085] D style =MLP s (Concat(Econtent , E style ))
[0086] Among them, D style Indicates the coordinate offset of the style landmark point, MLP s Represents the style encoder used to obtain the coordinate offset of the style landmark point, Concat represents E content and E style The connection function.
[0087] S43, the content marker point coordinate offset, the style marker point coordinate offset and the facial marker extracted from the reference facial image are combined. Add together to get the coordinates of the landmark points of the predicted image The formula involved is as follows:
[0088] L p =L0+D content +D style
[0089] For the landmark point coordinate prediction of the predicted image, the goal is to make the predicted landmark point coordinates as consistent as possible with the facial landmark point coordinates in the reference facial image. To this end, we define the landmark point loss function is the predicted value L of the landmark coordinates predict (L p ) and the true value L v The absolute error between them can be formally expressed as:
[0090]
[0091] S5. Construct a landmark point map based on the landmark point coordinates of the predicted image through the image-to-image transformation model The image-to-image transformation traverses the landmark graph and locates feasible input semantic points, so that the landmark graph consists of semantically connected lines between landmarks in the predicted image.
[0092] S6. Compare the landmark point map with the reference image in the reference facial image Superposition, using the generative network (U-Net) to generate the synthesized face image Compare the difference in pixel values between the synthesized face image and the real face image, and use this as the loss function as follows:
[0093]
[0094] in, represents the image loss function, I s represents the real face image, I t Represents a synthesized face image.
[0095] The embodiment of the present invention explicitly separates the speech content, emotional part, and identity characteristics of the speaker from the audio features in the voice audio, explicitly controls the facial expressions of the reference facial image based on the separated emotional part features, and synthesizes realistic and accurate facial animation videos in a more fine-grained manner.
[0096] An embodiment of the present invention also provides a facial animation synthesis system, which includes any preferred implementation of the above-mentioned method for separating emotions based on audio features and driving facial animation synthesis, as well as a combination of the preferred implementations without conflict. It should have the same or corresponding technical functions and beneficial effects as the method for separating emotions based on audio features and driving facial animation synthesis, which will not be repeated here.
[0097] The experimental results of this embodiment are as follows:
[0098] Dataset:
[0099] In order to evaluate the effectiveness of this method, experiments are conducted on the following public datasets.
[0100] VoxCeleb2, a large-scale audiovisual dataset of human speech, contains speeches from speakers of diverse ethnicities, accents, occupations, and ages. The videos were downloaded and edited from YouTube and contain over one million speeches by 6,112 celebrities. The faces in the videos vary in posture and other aspects, and the environments are also noisy, including background chatter, laughter, and varying lighting conditions.
[0101] Implementation details:
[0102] The network architectures of the MLP and LSTM are shown in Table 1. LeakyReLU and batch normalization are used after each hidden layer of the MLP. For the LSTM, a dropout of 0.2 is used to prevent overfitting of the neural network.
[0103] Table 1
[0104]
[0105]
[0106] Train the model: First use the loss function Training the content encoder MLP c , LSTM and MLP t ; Then use the loss function and Training the style encoder MLP e 、MLP i , self-attention encoder and MLPs ; Finally, use the loss function Train the face image generation part.
[0107] The open source deep learning framework PyTorch was used to implement the model. The adaptive moment estimation (Adam) optimizer was used during training, and the learning rate was set to 1e-5. The entire model was trained on an Nvidia 3090 GPU for nearly 60 hours.
[0108] Quantitative comparison:
[0109] Landmark distance (LMD) and expression similarity (ES) are used as evaluation indicators, where LMD is the Euclidean distance of 68 facial landmarks between the real face image and the synthesized face image. This indicator is used to evaluate whether the facial structure of the synthesized face image is close to the real face image. ES is the cosine similarity between the expression vector of the real face image and the expression vector of the synthesized face image. The expression vector is obtained through a pre-trained network. This evaluation indicator is used to evaluate whether the expression information of the synthesized face image is similar to the real face image. As shown in Table 2 below, the results show that on the VoxCeleb2 dataset, the method adopted by the present invention achieves the best performance in both LMD and ES indicators. In addition, the ablation experiments in the table show that the complete solution is significantly better than the solution without The baseline model of , demonstrates the effectiveness of contrastive learning of the style encoder. By comparing the two baselines, we can see that although the use of the style part improves the performance, its advantage cannot be fully exploited without contrastive learning.
[0110] Table 2
[0111]
[0112]
[0113] As shown in Table 2, MakeItTalk is a new deep learning-based architecture that can recognize facial landmarks, the chin, head pose, eyebrows, and nose, and can also change the lips in response to sound stimulation. The Pose-Controllable Audio-Visual System (PC-AVS) achieves free control of the human head pose within a framework of feature learning and image reconstruction.
[0114] Qualitative analysis:
[0115] In order to analyze the emotion space learned by the embodiment of the present invention, the following two qualitative experiments were conducted. The first qualitative experiment: the trained model was used to obtain the emotion feature embeddings of all training videos. Then, principal component analysis was performed on these emotion feature embeddings, and different emotion feature embeddings were sampled using the first principal component. Three videos were synthesized using the same audio, such as Figure 3A As shown, the video clips in the second and third rows are synthesized by adding the extracted emotional feature embedding values along the first principal component. The facial expressions of the three video clips change smoothly from solemn to excited, and the lip movements are gradually exaggerated.
[0116] Second qualitative experiment: The embodiment of the present invention visualizes the distance curve between the upper lip and the lower lip in three videos. Figure 3B As shown in the figure, Frame Number represents the frame number, and LIP distance represents the task-specific method. Different curves show similar trends, while lip distances vary significantly between videos. This result demonstrates that the present invention can learn diverse conversation styles while maintaining consistency between speech audio and facial expressions.
[0117] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0118] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.
Claims
1. A method for separating emotions based on audio features and driving facial animation synthesis, characterized in that: The steps include: S1. Given a speech audio segment to be animated and a dynamic reference facial image; S2, extracting audio features from the speech audio and facial landmarks from the reference facial image; S3, separating content information, identity information and emotion information from the extracted audio features combined with the reference facial image; S4, using an MLP fully connected neural network to combine the content information, identity information, emotion information and facial landmarks of the audio features to obtain the landmark coordinates of the predicted image; S5. Constructing a landmark point map according to the landmark point coordinates of the predicted image; S6, superimposing the landmark point map with the image in the reference facial image, and generating a synthesized facial image using a generative network; In step S3, the content information, identity information, and emotion information are separated from the extracted audio features in combination with the reference facial image, including the following steps: S31. Encode audio features using neural networks; S32. Using a long short-term memory network to capture the temporal information of the audio features, obtain content information including content feature embedding; S33, use two neural networks to convert audio features into emotion feature embedding and identity feature embedding respectively; S34, training a long short-term memory network and a neural network in combination with the reference facial image by contrastive learning to separate the emotion information embedded with the emotion feature and the identity information embedded with the identity feature; S35, inputting the emotion information including the emotion feature embedding and the identity information including the identity feature embedding into the self-attention encoder to obtain the style information including the style feature embedding; In step S34, the emotional information embedded with the emotional features and the identity information embedded with the identity features are separated by contrastive learning in combination with the reference facial image, including the following steps: S341, selecting a first video segment and a second video segment from the reference facial image, and then selecting a video segment of the same speaker as in the reference facial image as a third video segment; S342, performing comparative learning on the three video clips in a comparative learning model, and separating the emotional information including the emotional feature embedding according to the emotional feature embedding of the first video clip and the emotional feature embedding of the second video clip; S343, predicting the softmax probability of expressions in the three video clips by the expression prediction network; S344 , separating identity information including identity feature embedding according to the identity feature embedding of the first video segment and the identity feature embedding of the third video segment.
2. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 1, characterized in that: In step S2, audio features are extracted from the speech audio. First, the speech audio is preprocessed, and the preprocessed speech audio is extracted to obtain the number of audio frames of the audio features and the dimension of each frame of the audio features.
3. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 1, characterized in that: In step S2, facial landmarks are extracted from the reference facial image, comprising the following steps: S21, extracting all three-dimensional facial landmark coordinates from the reference facial image; S22, calculating the average coordinates of all three-dimensional facial landmark points; S23, aligning the coordinates of each 3D facial landmark point with the average coordinate value using an iterative closest point algorithm; S24. Project the three-dimensional facial landmark points onto the image space through orthogonal projection.
4. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 1, characterized in that: In step S342, the sentiment loss formula of contrastive learning is as follows: in, represents the sentiment loss function, is the cosine similarity between the softmax probabilities of the predicted expressions of the first and second video clips, represents the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the second video clip, The cosine similarity between the softmax probabilities of the predicted expressions of the first and third video clips, is the mean absolute error between the emotional feature embedding of the first video clip and the emotional feature embedding of the third video clip.
5. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 4 is characterized in that: In step S343, identity information including identity feature embedding is separated based on the identity feature embedding of the first video clip and the identity feature embedding of the third video clip. The identity loss formula of the contrastive learning is as follows: in, represents the identity loss function, represents the identity feature embedding of the first video clip, Represents the identity feature embedding of the third video clip.
6. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 5, characterized in that: In step S4, the coordinates of the landmark points of the predicted image are obtained by combining the content information, identity information, emotion information and facial landmark points of the audio features through a neural network, which specifically includes the following steps: S41, inputting the content information including the embedded content features into the neural network to obtain the content landmark point coordinate offset; S42, inputting the emotional information including the emotional feature embedding and the style information including the style feature embedding into the neural network to obtain the coordinate offset of the style landmark point; S43 , adding the content landmark point coordinate offset, the style landmark point coordinate offset, and the facial landmark points extracted from the reference facial image to obtain the landmark point coordinates of the predicted image.
7. The method for separating emotions based on audio features and driving facial animation synthesis according to claim 1, characterized in that: In step S6, after the synthesized facial image is generated by the generative network, the difference in pixel values between the synthesized facial image and the real facial image is compared. The image loss function is as follows: in, represents the image loss function, represents a real face image, Represents a synthesized face image.
8. A facial animation synthesis system, characterized in that: The method comprises the method for separating emotions based on audio features and driving facial animation synthesis as described in any one of claims 1 to 7.