A method, system, electronic device, and storage medium for generating facial expression animations.

Abstract

This invention relates to a method for generating facial expression animations, comprising: constructing an facial expression animation generation model, the model including a text feature encoder, a speech feature encoder, an expression decoder, and a linear layer; constructing a training dataset; training the facial expression animation generation model using the training dataset; acquiring corresponding text data and speech data; obtaining text features and speech features based on the text data and speech data; inputting the text features and speech features into the trained facial expression animation generation model to obtain 3D facial expression animation data; and redirecting the 3D facial expression animation data onto a target digital face to generate facial expression animations on the target digital face. Compared with existing technologies, this invention only requires speech data and text data to complete the generation of facial expression animations for the entire face, resulting in lower hardware costs and computational complexity. Moreover, by redirecting the 3D facial expression animation data onto the target digital face, the output is more flexible.

Description

Technical Field

[0001] This invention relates to the field of facial expression generation technology, and in particular to a multimodal driven facial expression animation generation method, system, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology and the emergence of the "metaverse" concept, facial expression generation technology for virtual digital humans has become a hot research topic, finding wide application in animation and film production, games, virtual reality, and human-computer interaction. Based on the input, facial expression generation can be divided into video-driven and voice / text-driven methods. Video-driven facial animation generation technology is relatively mature, while voice / text-driven facial animation generation technology, although proposed as early as the 1970s, has only seen rapid development in recent years with the advancement of artificial intelligence. Currently, generating more vivid and natural facial animations and non-mechanical expressions is a key research objective.

[0003] Common solutions involve either inputting only speech or only text. Speech input methods can be divided into two categories: one uses raw speech features as input, which relies on large amounts of training data and yields personalized results with weak generalization ability, requiring specific network parameters to be trained for specific individuals; the other uses an acoustic model trained on a large corpus for speech preprocessing, which loses a significant amount of intonation and emotional features. For text input methods, a common approach is to construct a one-to-one mapping table between text and facial expressions to obtain corresponding facial animations. However, the animations generated in this way are too static and lose emotional information. Another approach is to use speech synthesis technology to generate speech and then perform facial expression generation, which faces the same problems as inputting only speech.

[0004] In existing technologies, to improve the realism and vividness of animations, some researchers have considered multimodal input schemes, such as the Chinese invention patent applications CN202110718414.8 and CN202210744504.9, "A Method for Generating Emotion-Controllable Facial Animations Based on Multimodal Driving." These methods enhance the emotional expression of generated facial animations by inputting additional video information. The paper "Audio-and Gaze-driven Facial Animation of Codec Avatars" incorporates eye-tracking information to increase the expressiveness of upper facial expressions. However, these methods all require additional visual sensors, increasing hardware costs, and image processing consumes significant computational resources. In the Chinese invention patent application "Method for Synthesizing Fake Face Animations" with application number CN202110021844.4, a face sketch is generated by using multi-frame audio features from speech information and multi-frame prosodic features from text information, as well as an encoder-decoder network. On the one hand, this application uses prosodic features provided by speech synthesis frameworks such as Merlin. However, speech can most intuitively reflect prosodic features. Prosodic features synthesized from text may not match the original speech expression and may even have the opposite effect. On the other hand, its decoder only outputs the 2D key point motion of the lower half of the face. The features of the upper half of the face need to be obtained using the original video, which has high computational complexity, and the final output can only correspond to a specific input video.

[0005] Therefore, it is necessary to provide an expression generation scheme that has low hardware cost, low computational complexity, and flexible output. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a facial animation generation solution.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides a method for generating facial expression animations, comprising: constructing an facial expression animation generation model, the model including a text feature encoder, a speech feature encoder, an expression decoder, and a linear layer; the input of the model being text features and speech features, and the output being 3D facial expression animation data; constructing a training dataset; training the model using the training dataset to obtain a trained model; acquiring corresponding text data and speech data; obtaining text features and speech features based on the text data and speech data; inputting the text features and speech features into the trained model to obtain 3D facial expression animation data; and redirecting the 3D facial expression animation data onto a target digital face to generate facial expression animations on the target digital face.

[0009] Optionally, obtaining text features and speech features based on the text data and speech data specifically involves: aligning the text data and speech data at the phoneme level to obtain the phonemes corresponding to the speech at each time period, serializing the phonemes, and using them as text features; and extracting the Mel-frequency cepstral coefficients (MFCCs) of the speech data as speech features.

[0010] Optionally, the input of the text feature encoder is text features, the input of the speech feature encoder is the speech features and the output of the text feature encoder, the input of the expression decoder includes the output of the expression decoder at the previous moment and the connection between the output of the text feature encoder and the output of the speech feature encoder, the input of the linear layer is the output of the expression decoder, and the output of the linear layer is 3D expression animation data.

[0011] Optionally, the text feature encoder includes a first embedding layer and multiple stacked text feature encoding blocks; the input of the first embedding layer is text features, and the output of the first embedding layer is a text feature vector with added position encoding; the text feature encoding block includes a multi-head attention layer and a first fully connected layer, the input of the multi-head attention layer is a text feature vector with added position encoding, the output of the multi-head attention layer is fed into the first fully connected layer after residual concatenation and normalization, the output of the first fully connected layer is fed into the first fully connected layer after residual concatenation and normalization to obtain a text output vector, and the text output vectors of each text feature encoding block are serialized as the output of the text feature encoder.

[0012] Optionally, the speech feature encoder includes a second embedding layer and multiple stacked speech feature coding blocks; the input of the second embedding layer is speech features, and the output of the second embedding layer is a speech feature vector with added position encoding; the speech feature coding block includes a first cross-modal multi-head attention layer and a second fully connected layer; the input of the first cross-modal multi-head attention layer is the speech feature vector with added position encoding and the output of the text feature encoder; the output of the first cross-modal multi-head attention layer is fed into the second fully connected layer after residual connection and normalization; the output of the second fully connected layer is fed into the second fully connected layer after residual connection and normalization to obtain a speech output vector; the speech output vectors of each speech feature coding block are serialized and used as the output of the speech feature encoder.

[0013] Optionally, the expression decoder includes a third embedding layer and multiple stacked decoding blocks; the input of the third embedding layer is the output of the expression decoder at the previous time step, and the output of the third embedding layer is a first feature vector; the decoding block includes a linear bias multi-head attention layer, a second cross-modal multi-head attention layer, and a third fully connected layer; the input of the linear bias multi-head attention layer is the first feature vector; the output of the linear bias multi-head attention layer is fed into the second cross-modal multi-head attention layer after residual concatenation and normalization; the input of the second cross-modal multi-head attention layer also includes a concatenation of the output of the text feature encoder and the output of the speech feature encoder; the output of the second cross-modal multi-head attention layer is fed into the third fully connected layer after residual concatenation and normalization; the output of the third fully connected layer is fed into the third fully connected layer after residual concatenation and normalization to obtain the expression output vector; the expression output vectors of each decoding block are serialized and used as the output of the expression decoder.

[0014] Optionally, when training the facial animation generation model using the training dataset, the training aims to achieve convergence of the loss function, which is a weighted sum of vertex distance loss, temporal continuity loss, and mesh deformation loss.

[0015] Optionally, the real face mesh corresponding to the real 3D facial animation data is determined, and the predicted face mesh corresponding to the predicted 3D facial animation data is determined. The value of the vertex distance loss is the mean square error between the vertices of the real face mesh and the vertices of the predicted face mesh.

[0016] Optionally, the real face mesh corresponding to the real 3D facial animation data is determined, and the predicted face mesh corresponding to the predicted 3D facial animation data is determined. The value of the temporal continuity loss is the mean square error between the vertex displacements of adjacent frames of the real face mesh and the vertex displacements of adjacent frames of the predicted face mesh.

[0017] Optionally, the real face mesh corresponding to the real 3D facial animation data is determined, the predicted face mesh corresponding to the predicted 3D facial animation data is determined, and a preset neutral face mesh is obtained. The value of the mesh deformation loss is the mean square error between the deformation of the real face mesh relative to the neutral face mesh and the deformation of the predicted face mesh relative to the neutral face mesh.

[0018] Optionally, redirecting the 3D facial animation data to the target digital face includes:

[0019] Determine the predicted expression corresponding to the 3D facial animation data, obtain a set of pre-set expression bases, and obtain pre-set neutral expressions;

[0020] With the goal of minimizing the difference between the deviation value of each expression basis and the predicted expression, the coefficients of each expression basis are solved. The deviation value is the product of the coefficient of the expression basis and the correction value of the expression basis. The correction value is the difference between the expression basis and the neutral expression.

[0021] Obtain the neutral expression of the target digital face, combine the expression base and its coefficients with the neutral expression of the target digital face, and generate an expression animation on the target digital face.

[0022] Optionally, acquiring the corresponding text and voice data includes: acquiring text and voice data that are synchronized in time and have the same content.

[0023] Optionally, obtaining the corresponding text data and voice data includes: obtaining the text data and using a text-to-speech method to convert the text data into voice data.

[0024] Optionally, obtaining the corresponding text data and voice data includes: obtaining voice data and using a speech-to-text method to convert the voice data into text data.

[0025] This invention provides an expression animation generation system, comprising:

[0026] The model building module is used to build an expression animation generation model. The expression animation generation model includes a text feature encoder, a speech feature encoder, an expression decoder, and a linear layer. The input of the expression animation generation model is text features and speech features, and the output of the expression animation generation model is 3D expression animation data.

[0027] The training dataset building module is used to build the training dataset;

[0028] The training module is used to train the facial expression animation generation model using the training dataset to obtain a trained facial expression animation generation model.

[0029] The prediction module is used to acquire corresponding text data and speech data, acquire text features and speech features based on the text data and speech data, and input the text features and speech features into the trained facial expression animation generation model to obtain 3D facial expression animation data.

[0030] The redirection module is used to redirect the 3D facial animation data to the target digital face, and generate facial animation on the target digital face.

[0031] This invention provides an electronic device, including a memory and one or more processors. The memory stores a computer program, and the one or more processors are used to execute the computer program stored in the memory to implement the above-described facial animation generation method.

[0032] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the above-described facial animation generation method.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention does not require additional sensors; it can generate facial expression animations using only voice and text data. The data processing complexity of voice and text is also much lower than that of images. Furthermore, by redirecting 3D facial expression animation data to the target digital face, the output can be more flexible, meeting various facial expression animation generation needs, rather than being limited to a specific video. Attached Figure Description

[0035] Figure 1 A schematic diagram illustrating the application environment of the facial animation generation method;

[0036] Figure 2 A flowchart for the method of generating facial expression animations;

[0037] Figure 3 This is a schematic diagram of the overall structure of the facial expression animation generation model in Example 1;

[0038] Figure 4 This is a schematic diagram of the text feature encoder in Example 1;

[0039] Figure 5 This is a schematic diagram of the speech feature encoder in Example 1;

[0040] Figure 6 This is a schematic diagram of the facial expression decoder in Example 1;

[0041] Figure 7 A schematic diagram of an facial animation generation system;

[0042] Attached reference numerals: 100, Terminal; 1, Model building module; 2, Training dataset building module; 3, Training module; 4, Prediction module; 5, Redirection module. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, providing detailed implementation methods and specific operating procedures. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and the scope of protection of the present invention is not limited to the following embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] The term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. In the description of the invention, it should be understood that the terms "first," "second," and "third," etc., in the specification, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0045] This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or server products, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment), or the execution order of steps without timing restrictions can be adjusted.

[0046] Example 1

[0047] According to the background art, the technical problem to be solved by this application is to provide a facial animation generation solution with low hardware cost, low computational complexity and flexible output.

[0048] To address the aforementioned technical problems, this invention provides an embodiment of a facial expression animation generation method. As an optional implementation, this facial expression animation generation method can be applied to, but is not limited to, [examples of other methods]. Figure 1In the application environment shown, terminal 100 runs an expression animation editing application. In this application, corresponding text data and audio data are obtained, the target digital face is obtained, and 3D expression animation data is obtained based on the text data, audio data and tag animation generation model. The 3D expression animation data is essentially the vertex displacement of the 3D face at each time period. The 3D expression animation data is redirected to the target digital face, thereby generating expression animation on the target digital face.

[0049] The facial expression animation generation model includes a text feature encoder, a speech feature encoder, an facial expression decoder, and a linear layer. The input to the facial expression animation generation model is text features and speech features, and the output of the facial expression animation generation model is 3D facial expression animation data.

[0050] This invention does not require additional sensors; it can generate facial expression animations using only voice and text data. The data processing complexity of voice and text is also much lower than that of images. Furthermore, by redirecting 3D facial expression animation data to the target digital face, the output can be more flexible, meeting various facial expression animation generation needs, rather than being limited to a specific video.

[0051] Unlike prosodic features, the phoneme features used in this invention are directly related to mouth shape pronunciation, providing the most intuitive reflection of the original expression. This invention uses information from both speech and text modalities as input to the facial animation generation model. The original speech features retain information such as timbre, pitch, and emotion, while the text contains highly deterministic pronunciation information, i.e., phonemes. This avoids the uncertainty caused by the facial animation generation model extracting pronunciation information from speech. Fusing speech and text features significantly reduces the dependence on training data while producing higher accuracy and more vivid facial animations.

[0052] In this embodiment, the terminal may include, but is not limited to, at least one of the following: mobile phone, tablet computer, laptop computer, desktop PC, and other hardware devices used to generate facial animations. The above is merely an example, and this embodiment does not impose any limitations on it.

[0053] In this embodiment, the facial expression animation generation method can be applied, but is not limited to, applications that require editing facial expression animations. For example, it can be applied to the generation of facial expression animations for virtual characters in games and live streams. The virtual character's face can be used as the target digital face, and voice and text data from the game or live stream scene can be obtained to complete the generation of the virtual character's facial expression animation. The above is merely an example, and this embodiment does not impose any limitations on it.

[0054] According to this embodiment, a method for generating facial expression animations is provided, such as... Figure 2 As shown, it includes:

[0055] S1. Construct an facial expression animation generation model. The facial expression animation generation model includes a text feature encoder, a speech feature encoder, an facial expression decoder, and a linear layer. The input of the facial expression animation generation model is text features and speech features, and the output of the facial expression animation generation model is 3D facial expression animation data.

[0056] like Figure 3 As shown, the facial expression animation generation model contains two inputs and one output. Input1 is the text feature. Input2 is speech features The output is 3D facial animation data; the input to the text feature encoder is text features. The input to the speech feature encoder is speech features. and the output of the text feature encoder The input to the facial expression decoder includes h0, h1, ..., h t and Where h0, h1, ..., h t Output for text feature encoder With the output of the speech feature encoder The connection, The input of the linear layer is the output of the expression decoder at the previous time step; the input of the linear layer is the output of the expression decoder. The output of the linear layer is 3D facial animation data, which is the vertex displacement of the 3D face at each time interval.

[0057] like Figure 4 As shown, the text feature encoder includes a first embedding layer and multiple stacked text feature encoding blocks. The first embedding layer is a learnable embedding layer. To enable the network to capture temporal information, positional encoding is added to the features encoded by the embedding layer. The input of the first embedding layer is the text features, and the output of the first embedding layer is the text feature vector with added positional encoding. Each text feature encoding block contains two sub-layers: a multi-head attention layer and a fully connected layer (FC). The output of the multi-head attention layer is fed into the FC after residual concatenation and normalization. The output of the FC is then fed into the FC after residual concatenation and normalization to obtain the text output vector. The text output vectors of each text feature encoding block are serialized and used as the output of the text feature encoder. In this embodiment, the number of text feature encoding blocks is set to Na = 3, where Na represents the number of text feature encoding blocks (not fully shown in the figure). In other embodiments, those skilled in the art can adjust the number of text feature encoding blocks according to experience, resource consumption, and the needs of facial animation generation effects, such as 4 or 6.

[0058] likeFigure 5 As shown, the speech feature encoder includes a second embedding layer and multiple stacked speech feature encoding blocks. The second embedding layer is a learnable embedding layer. To enable the network to capture temporal information, positional encoding is added to the features encoded by the embedding layer. The input of the second embedding layer is the speech features, and the output of the second embedding layer is the speech feature vector with added positional encoding. Each speech feature encoding block contains two sub-layers: a first cross-modal multi-head attention layer and a second fully connected layer. The input of the first cross-modal multi-head attention layer is the speech feature vector with added positional encoding and the output of the text feature encoder. The output of the first cross-modal multi-head attention layer is fed into the second fully connected layer after residual concatenation and normalization. The output of the second fully connected layer is fed into the second fully connected layer after residual concatenation and normalization to obtain the speech output vector. The speech output vectors of each speech feature encoding block are serialized and used as the output of the speech feature encoder. In this embodiment, the number of speech feature coding blocks is set to Nb = 6, where Nb represents the number of speech feature coding blocks, which is not fully shown in the figure. In other embodiments, those skilled in the art can adjust the number of speech feature coding blocks according to experience, resource consumption, and the needs of facial animation generation, such as 4 or 7.

[0059] like Figure 6 As shown, the expression decoder includes a third embedding layer and multiple stacked decoding blocks. The third embedding layer is a learnable embedding layer, and its input is the output of the expression decoder at the previous time step. The output of the third embedding layer is the first feature vector. Each decoding block contains three sub-layers: a linear bias multi-head attention layer, a second cross-modal multi-head attention layer, and a third fully connected layer. The second cross-modal multi-head attention layer is a single cross-modal multi-head attention layer, and the third fully connected layer is a single fully connected layer. The input of the linear bias multi-head attention layer is the first feature vector. The output of the linear bias multi-head attention layer is fed into the second cross-modal multi-head attention layer after residual concatenation and normalization. The input of the second cross-modal multi-head attention layer also includes a concatenation of the outputs of the text feature encoder and the speech feature encoder. The output of the second cross-modal multi-head attention layer is fed into the third fully connected layer after residual concatenation and normalization. The output of the third fully connected layer is fed into the third fully connected layer after residual concatenation and normalization to obtain the expression output vector. The expression output vectors of each decoding block are serialized and used as the output of the expression decoder. In this embodiment, the number of decoding blocks is set to Nc = 6, where Nc represents the number of decoding blocks, which is not fully shown in the figure. In other embodiments, those skilled in the art can adjust the number of decoding blocks according to experience, resource consumption, and the needs of facial animation generation, such as 3 or 5.

[0060] The input to the third embedding layer is the output of the expression decoder at the previous time step. The output of the linear bias multi-head attention layer is fed into the second cross-modal multi-head attention layer after residual connection and normalization. The input of the second cross-modal multi-head attention layer also includes the connection between the output of the text feature encoder and the output of the speech feature encoder. Therefore, the expression decoder utilizes the outputs of the text feature encoder and the speech feature encoder, and also combines the outputs before the current time step, resulting in better continuity and more vivid expression animation.

[0061] The normalization process used in the text feature encoder, speech feature encoder, and facial expression decoder mentioned above is layer normalization (LN).

[0062] like Figure 3 As shown, the linear layer is a fully connected structure that projects the latent space representation of facial expressions output by the facial expression decoder into vertex motions of a 3D face.

[0063] Multi-head attention is an evolution of single-head attention, performing attention operations in groups (heads) to extract feature information from multiple dimensions. In a multi-head attention layer design, a given query and a series of key-value pairs are mapped together to an output. Text feature encoders include multi-head attention layers, where the query vector, key vector, and value vector all come from the output of the previous layer of the network.

[0064] The speech feature encoder and label decoder both include a cross-modal multi-head attention layer. Compared to a standard multi-head attention layer, the improvement of the cross-modal multi-head attention layer lies in the fact that its query and key vectors are not outputs of the previous layer. In the first cross-modal multi-head attention layer of the speech feature encoder, the query and key vectors come from the output of the text feature encoder, while the value vector comes from the output of the previous layer. In the second cross-modal multi-head attention layer of the expression decoder, the query and key vectors come from the outputs of both the text feature encoder and the speech feature encoder; that is, the outputs of the text feature encoder and the speech feature encoder are concatenated and then fed into the cross-modal multi-head attention layer of the expression decoder, while the value vector comes from the output of the previous layer.

[0065] The linear bias multi-head attention layer uses the ALiBi algorithm proposed in the paper "TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION". It masks information after the current time period, that is, it masks the information after the current time period in the output of the previous time period, and gives higher weights to features that are closer to the current time period.

[0066] This invention adds positional encoding to the speech and text features encoded by the Embedding layer. In this embodiment, sinusoidal positional encoding is used, which enables the network to capture temporal information.

[0067] In this invention, on the one hand, the text encoder and speech encoder are concatenated, which guides the feature extraction of the speech module, ignores the influence of noise, and makes the network more robust. On the other hand, the outputs of the text encoder and the speech encoder are combined as the input of the facial expression decoder. The features of the original speech retain information such as timbre, pitch, and emotion, while the text contains highly deterministic pronunciation information, avoiding the uncertainty caused by the network extracting pronunciation information from speech. The fusion of speech and text features can greatly reduce the dependence on the amount of training data, while producing higher accuracy and more vivid facial animations.

[0068] S2. Construct the training dataset;

[0069] Construct a training dataset to complement the facial animation generation model. Each sample includes text features, speech features, and the desired 3D face animation data. The 3D face animation data is described as the vertex displacement of the 3D face model at each time interval. Understandably, the text features and speech features should be aligned.

[0070] This embodiment uses an iPhone to collect 10 hours of training data, which includes voice data, text data and corresponding facial expressions. The facial expressions are converted into 3D facial animation data as the desired 3D facial animation data. The 3D facial mesh used contains 1220 vertices and 2304 faces.

[0071] In other implementations, corresponding voice and text data can be collected, a series of desired expressions can be generated using online expression editing software, and then 3D facial animation data corresponding to the expressions can be obtained.

[0072] First, a speech-text alignment algorithm is used to align the text and speech data in the collected training data at the phoneme level, obtaining the phonemes corresponding to each time segment of the speech. This embodiment uses the open-source speech-text alignment system (Montreal Forced Aligner, MFA) for text-speech alignment.

[0073] Then, audio and text features are extracted. In this application, the Mel-Frequency Cepstral Coefficients (MFCCs) of the speech data are used as speech features. In this embodiment, librosa is used for MFCC feature extraction, and the sampling rate is set to 16000Hz, the sliding window size is 0.02s, and the sliding window step size is 0.02s. Therefore, the frame rate of the extracted MFCC features is 50fps, that is, each frame contains 0.02s of speech signal. After obtaining the phonemes corresponding to each time period of speech, the phonemes are serialized and used as text features, that is, the phonemes are represented as category indices, and the category indices are used to represent the values ​​of the phonemes in the current time period.

[0074] S3. Use the training dataset to train the facial expression animation generation model to obtain the trained facial expression animation generation model;

[0075] The training dataset was divided into training, validation, and test sets in a 6:2:2 ratio to train the facial animation generation model. During training, the goal was to achieve convergence of the loss function, which was a weighted sum of vertex distance loss, temporal continuity loss, and mesh deformation loss, as follows:

[0076] Loss=ω1L dist +ω2L time +ω3L deform

[0077] Where Loss represents the value of the loss function, L dist L represents the vertex distance loss. time L represents the loss of temporal continuity. deform The value represents the mesh deformation loss, and ω1, ω2, and ω3 represent the weights of the vertex distance loss, temporal continuity loss, and mesh deformation loss, respectively.

[0078] ω1, ω2, and ω3 are used to weigh the proportions of the three parts of the loss function. In this embodiment, N = 1220, where N represents the number of vertices in the 3D face mesh, M = 2304, where M represents the number of faces in the 3D face mesh, ω1 = 1, ω2 = 5, and ω3 = 5. In other embodiments, those skilled in the art can adjust the values ​​of ω1, ω2, and ω3 according to experience and actual needs, and reset N and M.

[0079] Obtain the desired 3D face animation data from the training dataset to obtain its corresponding 3D face model / mesh, referred to as the real face mesh or real face or real face model. Obtain the 3D facial animation data output by the facial animation generation model to obtain its corresponding 3D face model / mesh, referred to as the predicted face mesh or predicted face or predicted face model. Pre-set a 3D face model / mesh with a neutral expression, referred to as the neutral face mesh or neutral face or neutral face model.

[0080] The vertex distance loss is the mean squared error between the vertices of the real face mesh and the vertices of the predicted face mesh, as follows:

[0081]

[0082] In the formula, L dist p represents the vertex distance loss. i t Let be the three-dimensional coordinates of the i-th vertex of the real face at time t. Let N be the number of vertices in the face model / mesh, where N is the predicted 3D coordinate of the i-th vertex of the face at time t.

[0083] The temporal continuity loss is the mean square error between the vertex displacements of adjacent frames of the real face mesh and the vertex displacements of adjacent frames of the predicted face mesh, as follows:

[0084]

[0085] In the formula, L time To account for temporal continuity loss, the mean square error between the displacement of vertices in the real face frames and the displacement of face vertices in the predicted face frames is calculated.

[0086] The mesh deformation loss is the mean square error between the deformation of the real face mesh relative to the neutral face mesh and the deformation of the predicted face mesh relative to the neutral face mesh, as follows:

[0087]

[0088] In the formula, L deform Indicates mesh deformation loss. Let t be the deformation of the real face relative to the neutral face at time t. Let p be the deformation of the face relative to a neutral face at time t. Let M be the number of triangles in the 3D face mesh. j1 ,p j2 ,p j3 and n j Let be the three vertices and normal vector of the j-th face of a neutral face. and Let be the three vertices and normal vector of the j-th face of the real face at time t. and Let be the three vertices and normal vector of the j-th face predicted at time t, then:

[0089]

[0090]

[0091] To improve the stability of network training, this embodiment calculates 50 PCA components on the vertex displacements of the 3D face animation data expected to be obtained in the training dataset, and uses them as the initial weight parameters of the linear layer of the multimodal driven facial animation generation model.

[0092] When the loss function stops decreasing and the error value stabilizes on the validation set, stop training, save the trained facial animation generation model, and verify the accuracy of the facial animation generation model on the test set. When the error is lower than the set threshold, the training of the facial animation generation model is complete; otherwise, continue training.

[0093] The loss function proposed in this invention not only calculates the error between the real vertex and the predicted vertex, but also constructs the deformation error between adjacent vertices of the 3D mesh. This can better simulate the subtle facial muscle changes in facial expressions, converge faster during training, and achieve higher accuracy.

[0094] S4. Obtain the corresponding text data and speech data, obtain text features and speech features based on the text data and speech data, input the text features and speech features into the trained facial animation generation model, and obtain 3D facial animation data.

[0095] Yes, it is understandable that text data and voice data should be corresponding, that is, synchronized in time and consistent in content. Obtaining corresponding text data and voice data includes obtaining text data and voice data that are synchronized in time and identical in content.

[0096] S5. Redirect the 3D facial animation data to the target digital face and generate facial animation on the target digital face.

[0097] To redirect 3D facial animation data onto a target digital human face, this application calculates facial coefficients for each time period based on a set of general facial bases, applies these coefficients to the target digital human face, and achieves facial expression-driven digital human expression control, including the following steps:

[0098] (1) Determine the predicted expression corresponding to the 3D facial animation data predicted by the facial animation generation model, obtain a set of preset expression bases, and obtain preset neutral expressions;

[0099] (2) With the goal of minimizing the difference between the deviation value of each expression base and the predicted expression, the coefficients of each expression base are solved. The deviation value is the product of the coefficient of the expression base and the correction value of the expression base, and the correction value is the difference between the expression base and the neutral expression.

[0100] The general expression base includes facial motion units defined by FACS. In this embodiment, the expression base uses Apple's ARKit Blendshapes specification, which contains 52 expression bases. This specification is widely applicable and highly compatible with various digital humans. The semantics of each expression base are shown in Table 1. Of course, in other implementations, other specifications can be used, such as NVIDIA's audio2face-defined Blendshapes and Faceware's Blendshapes, or a custom Blendshapes set can be defined.

[0101] Table 1 ARkit Hybrid Deformation List

[0102] 1.browDownLeft 19.eyeSquintLeft 36.mouthPressLeft 2.browDownRight 20.eyeSquintRight 37.mouthPressRight 3.browInnerUp 21.eyeWideLeft 38.mouthPucker 4.browOuterUpLeft 22.eyeWideRight 39.mouthRight 5.browOuterUpRight 23.jawForward 40.mouthRollLower 6.cheekPuff 24.jawLeft 41.mouthRollUpper 7.cheekSquintLeft 25.jawOpen 42.mouthShrugLower 8.cheekSquintRight 26.jawRight 43.mouthShrugUpper 9.eyeBlinkLeft 27.mouthClose 44.mouthSmileLeft 10.eyeBlinkRight 28.mouthDimpleLeft 45.mouthSmileRight 11.eyeLookDownLeft 29.mouthDimpleRight 46.mouthStretchLeft 12.eyeLookDownRight 30.mouthFrownLeft 47.mouthStretchRight 13.eyeLookInLeft 31.mouthFrownRight 48.mouthUpperUpLeft 14.eyeLookInRight 32.mouthFunnel 49.mouthUpperUpRight 15.eyeLookOutLeft 33.mouthLeft 50.noseSneerLeft 16.eyeLookOutRight 34.mouthLowerDownLeft 51.noseSneerRight 17.eyeLookUpLeft 35.mouthLowerDownRight 52.tongueOut 18.eyeLookUpRight

[0103] The expression coefficient is calculated by minimizing the following objective function:

[0104]

[0105] In the formula, B0 represents a neutral expression, B i (i∈[1,2,…,n]) represents the expression basis, n is the number of expression bases, and w i t ∈[0,1] represents the coefficient of the i-th expression basis in time period t. The predicted facial expression for time period t;

[0106] (3) Obtain the neutral expression of the target digital face, combine the expression base and its coefficients with the neutral expression of the target digital face, and generate an expression animation on the target digital face.

[0107] Any target digital face to which the emoji base is applied can be represented as:

[0108]

[0109] In the formula, T0 represents the neutral expression of the target digital human, and T i (i∈[1,2,…,n]) represents the facial expression basis of the target digital human, and F represents the facial expression of the target digital human at time t, where T i and B i Semantic correspondence, i.e., T i and B i They expressed the same expression.

[0110] By redirecting facial expressions, 3D facial animation data can be applied to different face models, offering good applicability and producing natural and vivid expressions. Other implementations can also use other methods to redirect 3D facial animation data to the target digital face, such as using machine learning algorithms or pre-setting animations corresponding to the vertices of the face mesh.

[0111] This invention also discloses an expression animation generation system, such as... Figure 7 As shown, it includes:

[0112] Model building module 1 is used to build an expression animation generation model. The expression animation generation model includes a text feature encoder, a speech feature encoder, an expression decoder, and a linear layer. The input of the expression animation generation model is text features and speech features, and the output of the expression animation generation model is 3D expression animation data.

[0113] Training dataset building module 2 is used to build the training dataset;

[0114] Training module 3 is used to train the facial animation generation model using the training dataset to obtain a trained facial animation generation model;

[0115] Prediction module 4 is used to acquire corresponding text data and speech data, obtain text features and speech features based on the text data and speech data, and input the text features and speech features into the trained facial expression animation generation model to obtain 3D facial expression animation data.

[0116] Redirection module 5 is used to redirect 3D facial animation data to the target digital face and generate facial animation on the target digital face.

[0117] The model building module is used to construct facial expression animation generation models. It includes a model library, a model selection unit, and a parameter tuning unit. The model library stores various neural network models. When constructing the facial expression animation generation model, the model selection unit selects one neural network model from the library or combines multiple neural network models to obtain the initial model. The model building module also has an input interface, allowing direct input of a single neural network model or a combination of neural network models to obtain the initial model. In the parameter tuning unit, parameters in the initial model can be manually adjusted, such as the number of network layers, initial weights, and the number of nodes in the input and output layers. Alternatively, parameter setting rules can be predefined, such as generating multiple facial expression animation generation models with different parameters given a dataset, and automatically completing the initial parameter settings by evaluating the performance of these models on the given dataset.

[0118] The training dataset construction module is used to build the training dataset. Its inputs are the corresponding speech and text data, as well as the desired 3D facial animation data. The training dataset construction module includes a training data acquisition unit, a data preprocessing unit, and a feature acquisition unit. The training data acquisition unit includes an input interface that can accept the collected training data, which includes speech data, text data, and corresponding facial expressions. In this unit, the facial expressions are converted into 3D facial animation data, which serves as the desired 3D facial animation data. The 3D facial mesh used contains 1220 vertices and 2304 faces. The training data acquisition unit can also include some expression editing applications, directly acquiring the corresponding speech and text data, generating a series of desired expressions using online expression editing software, and then obtaining the corresponding 3D facial animation data. The data preprocessing unit can perform noise reduction, data segmentation, and other operations. The feature acquisition unit is used to perform phoneme-level alignment of the speech and text data and extract speech and text features.

[0119] The training module is used to train the facial animation generation model using the training dataset. It allows setting the maximum number of training iterations, training termination conditions, and different performance evaluation metrics to evaluate the facial animation generation model after each training iteration to determine whether the model meets the requirements.

[0120] The prediction module includes an input interface for inputting corresponding text and speech data, and a feature acquisition unit for aligning speech and text data at the phoneme level and extracting speech and text features. Finally, the speech and text features are input into the facial animation generation model to obtain 3D face animation data output by the facial animation generation model.

[0121] The input to the redirection module is 3D face animation data and the target digital face. It sets up a variety of common expression bases, which can be selected from Apple ARKit's Blendshapes specification, NVIDIA's audio2face defined Blendshapes, Faceware's Blendshapes, and a custom set of Blendshapes. Then, the 3D face animation data is combined with the expression base to obtain expression coefficients, and then expression animation is generated on the target digital face through the expression coefficients.

[0122] This invention also discloses an electronic device, including a memory and one or more processors (e.g., a central processing unit (CPU)). The memory stores a computer program, and the one or more processors execute the computer program stored in the memory to implement the above-described facial animation generation method.

[0123] This invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed, implements the above-described facial animation generation method.

[0124] The aforementioned storage media may include computer-readable storage media such as non-volatile or non-transitory memory. Storage media may include ROM, RAM, magnetic disks, or optical disks.

[0125] Example 2

[0126] The network structure of the facial animation generation model can be adaptively adjusted. For example, in this embodiment, the facial animation generation module adopts a Transformer structure, which includes an encoder, a decoder and a linear layer. The encoder includes a speech feature encoder and a text feature encoder. The input of the facial animation generation model is text features and speech features, and the output of the facial animation generation model is 3D facial animation data.

[0127] The text feature encoder and the speech feature encoder have similar structures, both being the model architecture of the text feature encoder in Example 1. The difference is that the number of text feature coding blocks and speech feature coding blocks are set separately. Na=3 is set for the text feature encoder, and Nb=6 is set for the speech feature encoder. Both the text feature encoder and the speech feature encoder use a multi-head attention mechanism.

[0128] The structure of the decoder is the same as that of the facial expression decoder in Example 1. It uses a linear bias multi-head attention layer and a cross-modal multi-head attention layer. In the cross-modal multi-head attention layer, the query vector and the key vector come from the outputs of the text feature encoder and the speech feature encoder. That is, the outputs of the text feature encoder and the speech feature encoder are concatenated and then fed into the cross-modal multi-head attention layer of the facial expression decoder. The value vector comes from the output of the previous layer of the network.

[0129] In other implementations, the network structure of the facial animation generation model can also be designed using networks such as TCN and GRU as needed.

[0130] Example 3

[0131] In this embodiment, the loss function was adjusted and defined as: 1) the mean square error between the real face and the predicted face; 2) the mean square error between the 3D face vertex displacements of the real face and the predicted face in the preceding and following frames; and the weighted sum of 1) and 2).

[0132] In fact, the loss function in this embodiment is equivalent to the loss function in Embodiment 1 when the weight ω3 of the mesh deformation loss is 0.

[0133] In other implementations, those skilled in the art can design other loss functions according to their own needs, such as minimizing the difference between the vertices of the real face mesh and the vertices of the predicted face mesh.

[0134] Example 4

[0135] If it is not possible to obtain both text and voice data simultaneously when acquiring the corresponding data, then additional data is required, as follows:

[0136] If the original data only contains text data, then obtaining the corresponding text data and voice data includes: obtaining text data can be done through TTS technology, the AccessibilityService class in the Android system, text-to-speech methods based on machine learning models, etc., to convert text data into voice data.

[0137] If the original acquired data only contains voice data, then acquiring the corresponding text data and voice data includes: acquiring voice data can be done through STT technology, speech-to-text methods based on machine learning models, etc., to convert voice data into text data.

[0138] This invention relies solely on text and voice data to generate facial animations. The conversion between text and voice is a relatively mature existing technology. Therefore, this application has good scalability. In the case of only voice or only text, the corresponding text or voice can be obtained through voice-to-text or text-to-speech conversion, and it can still run stably.

[0139] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An expression animation generation method characterized by comprising: The method comprises the following steps: constructing an expression animation generation model, the expression animation generation model comprising a text feature encoder, a speech feature encoder, an expression decoder and a linear layer, an input of the expression animation generation model being text features and speech features, and an output of the expression animation generation model being 3D expression animation data; and constructing a training data set; training the expression animation generation model using the training data set to obtain a trained expression animation generation model; obtaining corresponding text data and speech data, obtaining text features and speech features based on the text data and the speech data, inputting the text features and the speech features into the trained expression animation generation model to obtain 3D expression animation data; and redirecting the 3D expression animation data to a target digital face to generate an expression animation on the target digital face; an input of the text feature encoder being text features, an input of the speech feature encoder being the output of the text feature encoder and speech features, an input of the expression decoder comprising an output of the expression decoder at a previous time and a connection of the output of the text feature encoder and the output of the speech feature encoder, an input of the linear layer being an output of the expression decoder, and an output of the linear layer being 3D expression animation data; the text feature encoder comprises a first embedding layer and a plurality of stacked text feature encoding blocks; an input of the first embedding layer is text features, and an output of the first embedding layer is a text feature vector to which position encoding is added; the text feature encoding block comprises a multi-head attention layer and a first full connection layer, an input of the multi-head attention layer is the text feature vector to which position encoding is added, an output of the multi-head attention layer is sent to the first full connection layer after residual connection and normalization processing, an output of the first full connection layer is obtained after residual connection and normalization processing, and a text output vector sequence of each text feature encoding block is serialized as an output of the text feature encoder.

2. The method of claim 1, wherein, The text features and the speech features are obtained based on the text data and the speech data in the following manner: the text data and the speech data are aligned at a phoneme level to obtain phonemes corresponding to speech at each time period, and the phonemes are serialized as text features; and mel-frequency cepstral coefficients (MFCCs) of the speech data are extracted as speech features.

3. The method of claim 1, wherein, the speech feature encoder comprises a second embedding layer and a plurality of stacked speech feature encoding blocks; an input of the second embedding layer is speech features, and an output of the second embedding layer is a speech feature vector to which position encoding is added; the speech feature encoding block comprises a first cross-modal multi-head attention layer and a second full connection layer, an input of the first cross-modal multi-head attention layer is the speech feature vector to which position encoding is added and the output of the text feature encoder, an output of the first cross-modal multi-head attention layer is sent to the second full connection layer after residual connection and normalization processing, an output of the second full connection layer is obtained after residual connection and normalization processing, and a speech output vector sequence of each speech feature encoding block is serialized as an output of the speech feature encoder.

4. The method of claim 1, wherein, The expression decoder comprises a third embedding layer and a plurality of stacked decoding blocks; an input of the third embedding layer is an output of the expression decoder at a previous moment, and an output of the third embedding layer is a first feature vector; the decoding block comprises a linear bias multi-head attention layer, a second cross-modal multi-head attention layer and a third fully connected layer; an input of the linear bias multi-head attention layer is the first feature vector; an output of the linear bias multi-head attention layer is sent to the second cross-modal multi-head attention layer after residual connection and normalization processing; an input of the second cross-modal multi-head attention layer further comprises a connection of an output of a text feature encoder and an output of a speech feature encoder; an output of the second cross-modal multi-head attention layer is sent to the third fully connected layer after residual connection and normalization processing; an output of the third fully connected layer is obtained after residual connection and normalization processing, and an expression output vector is obtained; and a sequence of the expression output vectors of each decoding block is taken as an output of the expression decoder.

5. The method of claim 1, wherein, When the expression animation generation model is trained using the training data set, the training is performed with the loss function converging as the target, and the loss function is a weighted sum of vertex distance loss, time continuity loss and mesh deformation loss.

6. The method of claim 5, wherein, The vertex distance loss value is the mean square error between the vertices of the real face mesh and the vertices of the predicted face mesh.

7. The method of claim 5, wherein, The time continuity loss value is the mean square error between the vertex displacements of the adjacent frames of the real face mesh and the vertex displacements of the adjacent frames of the predicted face mesh.

8. The method of claim 5, wherein, The mesh deformation loss value is the mean square error between the deformation of the real face mesh relative to the neutral face mesh and the deformation of the predicted face mesh relative to the neutral face mesh.

9. The method of claim 1, wherein, The 3D expression animation data is redirected to the target digital face, which comprises: determining a predicted expression corresponding to the 3D expression animation data, obtaining a group of pre-set expression bases, and obtaining a pre-set neutral expression; minimizing the difference between the bias value of each expression base and the predicted expression to solve the coefficient of each expression base, wherein the bias value is the product of the coefficient of the expression base and the correction value of the expression base, and the correction value is the difference between the expression base and the neutral expression; combining the neutral expression of the target digital face with the expression base and its coefficient to generate an expression animation on the target digital face.

10. The method of claim 1, wherein, The corresponding text data and speech data are obtained, which comprises: obtaining text data and speech data that are synchronous in time and identical in content.

11. The method of claim 1, wherein, The corresponding text data and speech data are obtained, which comprises: obtaining text data, and converting the text data into speech data using a text-to-speech method.

12. The method of claim 1, wherein, The obtaining of the corresponding text data and voice data comprises: obtaining voice data, and converting the voice data into text data by using a voice-to-text method.

13. An expression animation generation system characterized by comprising: The method comprises: a model construction module configured to construct an expression animation generation model, the expression animation generation model comprising a text feature encoder, a voice feature encoder, an expression decoder, and a linear layer, an input of the expression animation generation model being text features and voice features, and an output of the expression animation generation model being 3D expression animation data; a training data set construction module configured to construct a training data set; a training module configured to train the expression animation generation model by using the training data set to obtain a trained expression animation generation model; a prediction module configured to obtain corresponding text data and voice data, obtain text features and voice features based on the text data and the voice data, input the text features and the voice features into the trained expression animation generation model, and obtain 3D expression animation data; a redirection module configured to redirect the 3D expression animation data to a target digital face to generate an expression animation on the target digital face; the input of the text feature encoder is the text features, the input of the voice feature encoder is the output of the text feature encoder and the voice features, the input of the expression decoder comprises the output of the expression decoder at a previous time and the connection of the output of the text feature encoder and the output of the voice feature encoder, the input of the linear layer is the output of the expression decoder, and the output of the linear layer is the 3D expression animation data; the text feature encoder comprises a first embedding layer and a plurality of stacked text feature encoding blocks; the input of the first embedding layer is the text features, and the output of the first embedding layer is a text feature vector with position encoding added; the text feature encoding block comprises a multi-head attention layer and a first full connection layer; the input of the multi-head attention layer is the text feature vector with position encoding added, the output of the multi-head attention layer is sent to the first full connection layer after residual connection and normalization processing, the output of the first full connection layer is obtained after residual connection and normalization processing, and the text output vectors of each text feature encoding block are serialized to serve as the output of the text feature encoder.

14. An electronic device, comprising: The device comprises a memory and one or more processors, the memory has a computer program stored therein, and the one or more processors are configured to execute the computer program stored on the memory to implement the expression animation generation method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, A computer program is stored on the device, and the computer program is executed to implement the expression animation generation method according to any one of claims 1-12.

Images