Audio-driven facial animation generation method and apparatus

Abstract

The embodiment of the specification provides a kind of audio-driven face animation generation method and device.Method includes: audio segment is encoded as the audio feature sequence of the audio feature of multiple time frames;In sequence, for any target time frame of the multiple time frames, first prediction operation is executed, wherein it includes: using the encoder in auto-encoding system, the face data of h frames before target time frame is encoded, to obtain first feature;Using conditional predictor, first condition data is generated based on the audio feature sequence, identity feature representing speaker identity and the first feature, using diffusion model, latent representation is recovered from noise based on the first condition data;Using the decoder in the auto-encoding system, the latent representation is decoded, to obtain the face data of the target time frame.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of machine learning, and more particularly to an audio-driven facial animation generation method and apparatus. Background Technology

[0002] In recent years, audio-driven facial animation (such as 3D facial animation) generation has received considerable attention. The demand for realistic and synchronized 3D facial expression animations generated in real-time based on audio is growing in fields such as advertising, virtual reality, gaming, and telecommunications. Faced with this increasing demand, improving the quality of facial animation generation is a pressing issue that needs to be addressed.

[0003] Therefore, there is a need for a high-performance audio-driven facial animation generation solution. Summary of the Invention

[0004] This specification describes one or more embodiments of an audio-driven facial animation generation scheme that dynamically integrates historical facial data through an autoregressive approach to generate facial animations synchronized with audio input in real time, resulting in good performance when generating facial animations of arbitrary length.

[0005] According to a first aspect, an audio-driven facial animation generation method is provided, comprising: encoding an audio segment into an audio feature sequence composed of audio features of multiple time frames; sequentially performing a first prediction operation for any target time frame of the multiple time frames, including: encoding facial data of h frames prior to the target time frame using an encoder in an autoencoder system to obtain a first feature; generating first conditional data using a conditional predictor based on the audio feature sequence, identity features representing the speaker's identity, and the first feature; recovering a latent representation from noise using a diffusion model based on the first conditional data; and decoding the latent representation using a decoder in the autoencoder system to obtain facial data of the target time frame.

[0006] According to one embodiment, when the target time frame belongs to the first h time frames among the plurality of time frames, the facial data of the h frames before the target time frame is obtained by initialization.

[0007] According to one embodiment, the autoencoder system has a codebook containing primitive vectors of N facial primitives; encoding h frames of facial data prior to the target time frame to obtain a first feature includes: using the encoder to encode the h frames of facial data into a feature matrix composed of h feature vectors; reshaping the feature matrix into encoding units of multiple facial components; and mapping each encoding unit to the nearest primitive vector in the codebook to obtain the first feature.

[0008] In one embodiment, a conditional predictor generates first conditional data based on the audio feature sequence, the identity feature representing the speaker's identity, and the first feature. This includes: aggregating the first feature and the identity feature; inputting the aggregated first input sequence into a first attention module for self-attention processing to obtain a first output sequence; inputting the first output sequence and the audio feature sequence into a second attention module for cross-modal cross-attention processing to obtain a second output sequence; and inputting the second output sequence into a feedforward neural network for processing to obtain the first conditional data.

[0009] In a further embodiment, the self-attention processing includes: calculating attention scores for each position in the first input sequence, only for the elements of that position and its preceding positions, to obtain a causal attention matrix; and calculating a first output sequence based on the causal attention matrix.

[0010] In one embodiment, recovering a latent representation from noise using a diffusion model based on the first conditional data includes: performing multiple rounds of denoising, each round of denoising including: inputting the current noise, the first conditional data, and round features characterizing the current denoising round into a diffusion head for processing to obtain the latent representation of the current round; performing at least one diffusion process on the latent representation of the current round, and using the diffusion-processed latent representation as the current noise in the next round of denoising.

[0011] In a further embodiment, the diffuser head is a multilayer sensor.

[0012] According to one embodiment, the latent representation is decoded using a decoder in the autoencoder system to obtain facial data of the target time frame, including: mapping each feature vector in the latent representation to the nearest primitive vector in the codebook to obtain quantized latent features; and inputting the quantized latent features into the decoder for processing to obtain facial data of the target time frame.

[0013] According to one embodiment, the method further includes: generating a facial animation corresponding to the audio segment based on facial data from the plurality of time frames.

[0014] In one embodiment, the facial data includes the position information of V vertices in a facial mesh template.

[0015] In a further embodiment, the audio segment is taken from training samples, which also include real facial data corresponding to the audio segment; the method further includes: updating the conditional predictor and the diffusion model in the direction of decreasing training loss, wherein the training loss includes latent loss and vertex loss, the latent loss being used to characterize the difference between the latent representation and the real latent representation corresponding to the real facial data, and the vertex loss being used to characterize the difference between the predicted vertex state in the generated facial data and the real vertex state in the real facial data.

[0016] In a further embodiment, the vertex loss includes vertex position loss and vertex velocity loss, wherein the vertex position loss represents the distance between the generated vertex position and the real vertex position, the generated vertex position represents the vertex position determined based on the generated facial data, the real vertex position represents the vertex position determined based on the real facial data, the vertex velocity loss represents the difference between the generated vertex velocity and the real vertex velocity, the generated vertex velocity represents the distance between vertices in two adjacent frames of generated facial data, and the real vertex velocity represents the distance between vertices in two adjacent frames of real facial data.

[0017] According to a second aspect, an audio-driven facial animation generation apparatus is provided, comprising: an encoding unit configured to encode an audio segment into an audio feature sequence consisting of audio features of multiple time frames; and an execution unit configured to sequentially perform a first prediction operation on any target time frame of the multiple time frames, including: encoding facial data of h frames prior to the target time frame using an encoder in an autoencoder system to obtain a first feature; generating first conditional data using a conditional predictor based on the audio feature sequence, identity features representing the speaker's identity, and the first feature; recovering a latent representation from noise using a diffusion model based on the first conditional data; and decoding the latent representation using a decoder in the autoencoder system to obtain facial data of the target time frame.

[0018] According to a third aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0019] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0020] According to a fifth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.

[0021] In the embodiments of this specification, for a given audio segment, a limited number of past frames are selected as historical facial contexts and combined with the audio input to create conditions that guide the diffusion process to generate facial sequences in real time, overcoming the performance bottleneck of existing methods when generating facial sequences of arbitrary length. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating the principle of a facial animation generation method according to one embodiment is shown.

[0024] Figure 2 A flowchart of an audio-driven facial animation generation method according to one embodiment is shown.

[0025] Figure 3 A schematic diagram illustrating the working principle of a self-encoding system according to one embodiment is shown.

[0026] Figure 4 A schematic diagram of a facial data generation process according to one embodiment is shown.

[0027] Figure 5 A schematic diagram of an audio-driven facial animation generation apparatus according to one embodiment is shown. Detailed Implementation

[0028] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0029] In the description of this specification, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in this specification should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0030] In this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0031] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments in this specification are intended to be based on a given audio segment a 1:T =(a1,…,a T and speaker identity (style) k Generate the corresponding facial mesh sequence Among them, a i It is audio segment a 1:T The audio data corresponding to the i-th time frame. It is the facial mesh generated corresponding to the audio data of the i-th time frame, s k It can be one-hot encoding.

[0033] In some embodiments of this specification, facial data is used to represent facial motion, where the facial data is based on a facial mesh. Specifically, a facial mesh template can be pre-defined, including V vertices. The facial data for a given frame may include the position information of the V vertices in three-dimensional space at the corresponding moment of that frame (e.g., offset relative to the standard position of the template, or absolute position in three-dimensional space). Based on the position information of the V vertices represented by the facial data, the position information of the corresponding vertices in the facial mesh template can be adjusted to obtain the facial mesh representing the facial motion. Therefore, in some embodiments of this specification, the task of generating a facial mesh sequence (i.e., facial animation) can be converted into a task of generating a facial data sequence.

[0034] In recent years, diffusion models have achieved significant success in image generation tasks. Among related techniques, it has been proposed to apply diffusion models to audio-driven facial animation generation to improve facial generation results to some extent. However, this approach denoises the entire sequence at once, generating a facial sequence corresponding to a single audio segment. This approach is limited to generating fixed-length facial sequences. Performance degrades significantly when the length of the required facial sequence exceeds the training range. Furthermore, this approach requires the model to process the entire audio segment before outputting the result, making it difficult to achieve real-time response to dynamic audio input.

[0035] When the required facial sequence is long, i.e., when the audio segment to be processed is long, generating a facial sequence corresponding to the entire audio segment all at once using the above method yields poor results. If the long sequence generation task is broken down into multiple short sequence generation tasks, the generated facial sequences often lack coherence, and the sequence connections are discontinuous.

[0036] In view of this, this specification proposes an audio-driven facial animation generation method based on an autoregressive diffusion model. This method dynamically integrates historical facial data through an autoregressive approach to generate facial animations synchronized with audio input in real time, thereby overcoming the performance limitations of existing methods in generating facial animations of arbitrary length.

[0037] Figure 1 A schematic diagram illustrating the principle of a facial animation generation method according to one embodiment is shown.

[0038] like Figure 1 As shown, for the target time frame to be processed, several frames of historical facial data preceding the target time frame can be selected, and combined with audio data and speaker identity, a conditional predictor is used to generate conditions for predicting facial data for the target time frame. The generated conditions can guide the diffusion model to generate facial animation for the target time frame.

[0039] like Figure 1 As shown by the dashed lines, facial data reconstructed from the target time frame can be dynamically integrated into historical facial data to predict the conditions corresponding to the next time frame, thus participating in the generation of the next frame's facial data. In this way, this autoregressive diffusion processing method overcomes the performance limitations of existing methods in generating facial meshes of arbitrary length. Furthermore, the generation of each frame of facial data depends on several previous frames of historical facial data, ensuring the coherence between the generated facial data. In addition, this method supports real-time output of facial animation corresponding to dynamic audio input, making it particularly suitable for applications requiring real-time interaction, such as virtual reality, games, and remote communication.

[0040] The following describes the specific implementation process of the above technical concept.

[0041] Figure 2 A flowchart of an audio-driven facial animation generation method according to one embodiment is shown.

[0042] It should be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Furthermore, the model architecture corresponding to the technical concept of this method will utilize [the relevant technologies / methods] during both the training and usage phases. Figure 2 The forward reasoning process shown is used for prediction.

[0043] In step S21, the audio segment is encoded into an audio feature sequence composed of audio features from multiple time frames.

[0044] During the usage phase, the audio clip can be a piece of real-time acquired audio data, and the facial data corresponding to the audio clip is unknown. During the training phase, the audio clip can be taken from the training samples, and the training samples also include the real facial data corresponding to the audio clip.

[0045] An audio segment consists of audio data from multiple time frames. The audio data of each time frame in an audio segment can be encoded to obtain an audio feature sequence composed of audio features from multiple time frames.

[0046] Self-supervised pre-trained speech models outperform traditional feature extraction methods (such as Mel-frequency cepstral coefficients, MFCC) in audio feature extraction. Therefore, in some implementations, self-supervised pre-trained speech models can be used to encode audio segments into audio feature sequences. Self-supervised pre-trained speech models can be, but are not limited to, the HuberT model.

[0047] For example, the encoded audio feature sequence can be denoted as E. a (a)∈R T×C 'a' represents an audio segment, and 'E' represents an audio clip. a (a) represents the audio feature sequence obtained by audio encoding an audio segment, where C represents the dimension of the audio feature and the number of audio features is equal to the number of time frames T.

[0048] In step S22, the first prediction operation is performed sequentially for any target time frame of the multiple time frames.

[0049] For a given audio segment consisting of audio data from multiple time frames, the embodiments of this specification aim to sequentially generate facial data corresponding one-to-one with these multiple time frames in an autoregressive manner. Therefore, a time frame can be sequentially selected from the multiple time frames as the target time frame, and a first prediction operation can be performed on each selected target time frame.

[0050] The first prediction operation is used to generate facial data corresponding to the target time frame.

[0051] like Figure 2 As shown, the first prediction operation includes steps S231 to S237.

[0052] In step S231, the encoder in the self-encoding system is used to encode the facial data of the h frames prior to the target time frame to obtain the first feature.

[0053] As mentioned earlier, a frame of facial data includes the position information of V vertices in the facial mesh template at the corresponding time. The position information mentioned here can refer to three-dimensional coordinates or offsets relative to the standard positions of the vertices in the facial mesh template.

[0054] h can be a fixed value. A fixed history length strategy can be used to determine the facial data of h frames prior to the target time frame. For example, a starting time frame can be selected based on the target time frame, and the facial data corresponding to the starting time frame and the next few time frames can be used as the facial data of h frames prior to the target time frame.

[0055] To use T t Taking the target time frame as an example, the starting time frame T s =max(T) t -h,1), target time frame T t The previous h-frame facial data can be denoted as or Where x can represent real facial data, This can represent the facial data obtained by performing the first prediction operation. During the training phase, h frames of facial data can be h frames of real facial data or h frames of generated facial data. During the usage phase, the real facial data is unknown; therefore, h frames of facial data are defined as follows: for the target time frame T... t Historical facial data generated by performing the first prediction operation on previous time frames. If the length of the currently generated historical facial data is less than h, the entire historical facial data sequence can be selected. In this case, several frames of facial data can be initialized to pad the length of the historical predicted facial data to h.

[0056] In some embodiments, when the target time frame belongs to the first h time frames among multiple time frames, the facial data of the h frames preceding the target time frame (such as some or all of the facial data therein) can be facial data obtained through initialization. When the target time frame does not belong to the first h time frames among multiple time frames, the facial data of the h frames preceding the target time frame is historical facial data obtained by performing the first prediction operation on the h time frames preceding the target time frame (such as the h time frames closest to the target time frame).

[0057] An autoencoder (AE) system includes an encoder and a decoder. The encoder encodes the input facial data into a low-dimensional latent vector. The decoder reconstructs the original facial data from the low-dimensional latent vector. Both the encoder and decoder can be implemented using neural networks. Accordingly, in some embodiments, in this step, the neural network corresponding to the encoder can be used to encode facial data from h frames prior to the target time frame to obtain the first feature.

[0058] In some embodiments, the self-coding system may also include a codebook. Figure 3 A schematic diagram illustrating the working principle of a self-encoding system according to one embodiment is shown.

[0059] like Figure 3 As shown, the self-encoding system includes an encoder, a decoder, and a codebook.

[0060] The codebook contains primitive vectors for N facial primitives. For example, the codebook can be denoted as Z. Among them, z j Let represent the j-th primitive vector, and C be the dimension of the primitive vector.

[0061] The encoding process of a self-encoding system is as follows.

[0062] First, facial data x 1:T Encoded into continuous feature vectors by the encoder Continuous eigenvectors This refers to a feature matrix composed of T feature vectors. During the training phase, facial data x... 1:T This refers to the actual facial data in a T-frame. Facial data x 1:T Encoded into continuous feature vectors by the encoder The process can be recorded as follows: Where C represents the dimension of the feature vector, and the number of feature vectors is equal to the number of frames T of facial data input to the encoder.

[0063] Next, continuous eigenvectors (i.e., the feature matrix) is reshaped into encoding units for multiple facial components. Where H represents the number of facial components.

[0064] Finally, through quantization, each coding unit is mapped to the nearest primitive vector in the codebook, resulting in the quantized feature vector z. q , z q ∈R T′×H×C The quantization operation refers to calculating the distances between each coding unit and each primitive vector in the codebook, and selecting the nearest primitive vector as the quantization feature vector z for that coding unit. q For example, the quantized feature vector is represented as:

[0065]

[0066] in, express In the i-th encoding unit, quantize() represents the quantization operation, and |||2 represents the L2 norm.

[0067] The decoding process in an autoencoder system refers to using a decoder to reconstruct facial data from quantized feature vectors, completing the transformation from the latent space to the original space. The decoding process can be exemplified as follows: in, This represents the reconstructed facial data obtained after decoding, for example... Figure 3 As shown, this could refer to the original facial data x. 1:T Corresponding reconstructed facial data from the first time frame to the Tth time frame

[0068] by Figure 3 Taking the autoencoder system shown as an example, the process of encoding h frames of facial data using the encoder in the autoencoder system can be summarized as follows: First, the encoder encodes the h frames of facial data into a feature matrix composed of h feature vectors (corresponding to the continuous feature vectors mentioned above); then, the feature matrix is ​​reshaped into encoding units for multiple facial components; next, each encoding unit is mapped to the nearest primitive vector in the codebook to obtain the first feature. The first feature can be regarded as a feature sequence composed of multiple quantized feature vectors (primary vectors after mapping of encoding units).

[0069] In step S233, the conditional predictor generates first conditional data based on the audio feature sequence, the identity features representing the speaker's identity, and the first feature.

[0070] In the embodiments of this specification, speaker identity and speaker style are used interchangeably. Speaker identity (or speaker style) can be represented by a one-hot code. Identity features can be viewed as the projection result of the one-hot code into the latent space. For example, a simple linear projection layer can be used to embed the one-hot code representing the speaker identity into the latent space to obtain identity features. Identity features can be denoted as E. s (s k )∈R 1×C s k E is a one-hot encoding used to represent the speaker's identity. s (s k ) to s k The identity features obtained after encoding, where C is the dimension of the identity features.

[0071] In some embodiments, the conditional predictor can be implemented as a neural network that receives an audio feature sequence, a first feature, and an identity feature as input, and outputs first conditional data after processing by the neural network. Specifically, the conditional predictor can employ a type of neural network suitable for sequence processing, such as an LSTM neural network, a neural network based on transformer attention processing, etc.

[0072] Figure 4 A schematic diagram illustrating a facial data generation process according to one embodiment is shown below. Figure 4 In a specific example, the first conditional data C is explained. A The detailed generation process.

[0073] like Figure 4 As shown, the first feature and the identity feature can be aggregated. For example, the identity feature can be aggregated with each feature vector (such as the quantized feature vector) in the first feature (such as vector addition).

[0074] Next, the aggregated first input sequence can be input into the first attention module for self-attention processing to obtain the first output sequence. In some embodiments, self-attention processing refers to calculating attention scores for each position in the first input sequence only for that position and the elements of the positions preceding it, obtaining a causal attention matrix, and calculating the first output sequence based on the causal attention matrix. This ensures that only current or past information can be accessed during self-attention processing, preventing information leakage. The first attention module can employ, but is not limited to, a biased causal multi-head self-attention layer.

[0075] like Figure 4 As shown, before inputting the first input sequence into the first attention module, each sequence element in the first input sequence can be aggregated with its corresponding positional encoding (e.g., vector addition). The aggregated first input sequence is then input into the first attention module. Thus, by introducing positional information into the first input sequence, the problem of the self-attention mechanism's insensitivity to sequence order can be solved.

[0076] Then, the output of the first attention module (i.e., the first output sequence) and the audio feature sequence are jointly input into the second attention module for cross-modal cross-attention processing to obtain the second output sequence. Through cross-modal cross-attention processing, the first output sequence and the audio feature sequence can be aligned. For example, the second attention module can employ, but is not limited to, a biased cross-modal multi-head attention layer to better align the first output sequence and the audio feature sequence.

[0077] Finally, the second output sequence is input into the feedforward neural network for processing to obtain the first conditional data C. A .

[0078] In step S235, the latent representation is recovered from the noise using a diffusion model based on the first conditional data.

[0079] First conditional data C A This is used to guide the diffusion generation process. The diffusion model includes a forward process and a backward process. The forward process refers to progressively adding Gaussian noise to the initial latent variables through the DDPM (Denoising Diffusion Probabilistic Models) scheduler, transforming the original samples into a standard Gaussian distribution. The backward process refers to recovering the samples (i.e., the latent representation) from the noise through the conditional distribution. In the usage phase after the diffusion model has been trained, it is only necessary to recover the latent representation from the noise through the backward process. Considering that this conditional distribution depends on the entire dataset and is difficult to process directly, the embodiments in this specification propose that a diffusion head can be trained to approximate the conditional distribution of the backward process. In this way, accuracy can be maintained while ensuring real-time generation speed. For example, the diffusion head can be a multilayer perceptron (MLP).

[0080] In some embodiments, the diffusion head (e.g., MLP) can directly recover clean samples (i.e. latent representations) from noise based on first conditional data to provide more accurate facial constraints.

[0081] The following reference Figure 4 The process of recovering a potential representation from noise is illustrated by way of example.

[0082] See Figure 4 It can perform multiple rounds of noise reduction.

[0083] Each round of denoising includes: inputting the current noise, the first conditional data, and the round features representing the current denoising round into a diffusion head for processing to obtain the latent representation of the current round; performing at least one diffusion process on the latent representation of the current round, and using the diffused latent representation as the current noise in the next round of denoising. The diffusion process mentioned here refers to adding noise to the latent representation. The latent representation obtained in the final round of denoising is used for decoding.

[0084] In step S237, the latent representation is decoded using the decoder in the autoencoder system to obtain the facial data of the target time frame.

[0085] Taking an autoencoder system including a codebook as an example, quantizing the latent representation using the codebook results in discrete mappings, which increases the complexity of backpropagation. Therefore, in some embodiments, the latent representation can be directly input into the decoder of the autoencoder system, where the decoder decodes the latent representation to obtain the facial data of the target time frame. As an alternative, each feature vector in the latent representation can be mapped to the nearest primitive vector in the codebook to obtain quantized latent features, which are then input into the decoder for processing to obtain the facial data of the target time frame.

[0086] Thus, after step S23 is completed, facial data from multiple time frames can be obtained. Based on this facial data from multiple time frames, a facial animation corresponding to the audio segment can be generated.

[0087] Taking the facial data of each time frame, including the offset position information of V vertices in the facial mesh template relative to the standard position, as an example, for each of the multiple time frames, the positions of the V vertices in the facial mesh template can be adjusted based on the offset position information in the facial data corresponding to that time frame, thus obtaining the facial mesh corresponding to that time frame. The sequence of facial meshes corresponding to multiple time frames constitutes the facial animation corresponding to the audio segment.

[0088] The following is an example of the loss function used in the training process.

[0089] The entire training process can be divided into two stages: Stage 1 at the beginning and Stage 2 at the end.

[0090] Phase 1 refers to training the autoencoder system using real facial data.

[0091] The autoencoder system used in Phase 2 can be the same autoencoder system trained in Phase 1. In Phase 2, the parameters of the autoencoder system can remain unchanged, while only other model parts are trained. For example, in Phase 2, training samples including audio clips and corresponding real facial data can be used to train the conditional predictor and the diffusion model.

[0092] Phase 1 comes first, followed by Phase 2. Therefore, training an autoencoder system can be viewed as pre-training the autoencoder system. The loss function used to train the autoencoder system can include two parts: reconstruction loss and quantization loss.

[0093] Reconstruction loss is used to characterize the difference between the reconstructed facial data sequence and the real facial data sequence, which can be represented by the L1 distance between the reconstructed facial data sequence and the real facial data sequence.

[0094] The formula for calculating reconstruction loss can be exemplarily expressed as follows: Among them, L recIndicates the losses incurred during reconstruction. represents the reconstructed facial data (such as a reconstructed facial data sequence), x represents the real facial data (such as a real facial data sequence), and |||1 represents the 1-norm.

[0095] The quantization loss, which includes two intermediate code-level losses, is used to reduce the distance between the codebook and the embedded features.

[0096] The formula for calculating quantization loss can be exemplarily expressed as follows: Here, β is a weight hyperparameter used to control the update speed of the codebook and encoder; the value of β can be, for example, 0.25. sg() represents the gradient stopping operation, which means that the partial derivatives are zero during backpropagation. This represents the feature matrix obtained by the encoder from the original facial data sequence. q The quantized feature matrix is ​​obtained by mapping the eigenvectors in the feature matrix to the nearest primitive vector in the code.

[0097] The training loss in stage 1 can be denoted as:

[0098] L stage1 =λ rec L rec +λ quant L quant ,

[0099] Where, λ rec , λ quant For example, λ is the weight. rec , λ quant The value of can be 1.

[0100] During Phase 1 training, the training loss L in Phase 1 can be calculated. stage1 The direction of reduction involves updating the autoencoder system, such as updating the encoder, codebook, and decoder.

[0101] For autoencoder systems that do not contain a codebook, the quantization loss mentioned above can be removed, and only the reconstruction loss can be used for training.

[0102] During Phase 2 training, the conditional predictor and the diffusion model can be updated in the direction that reduces the Phase 2 training loss. The Phase 2 training loss can include latent loss and vertex loss.

[0103] The latent loss represents the distance between the generated latent representation and the true latent representation corresponding to the real facial data. The true latent representation can be obtained by encoding the real facial data using the encoder in the stage 1 trained autoencoder system, and mapping each feature vector in the encoded feature matrix to the nearest primitive vector in the codebook. Among them, L latent Indicates potential losses. The latent representation to be generated, z q It represents the true potential representation.

[0104] Vertex loss characterizes the difference between the predicted vertex states in the generated face data and the actual vertex states in the real face data. For example, vertex loss may include vertex position loss and vertex velocity loss.

[0105] Vertex position loss characterizes the distance between vertex positions in the generated facial data and vertex positions in the real facial data. For example, the formula for calculating vertex position loss can be expressed as follows: in, represents the generated facial data, and x represents the actual facial data. This represents the difference between the vertex positions in the generated facial data and the corresponding vertex positions in the real facial data.

[0106] Vertex velocity loss characterizes the difference between generated vertex velocities and real vertex velocities. Generated vertex velocity characterizes the distance between vertices in two adjacent frames of generated face data. Real vertex velocity characterizes the distance between vertices in two adjacent frames of real face data. For example, the formula for calculating vertex velocity loss can be expressed as follows:

[0107]

[0108] Among them, the first term in the above formula This represents the distance between the vertices of two adjacent frames of facial data generated; the second term in the formula. This represents the distance between vertices of two adjacent frames of real facial data.

[0109] In some embodiments, the training loss of phase 2 can be expressed as:

[0110] L stage2 =λ latent L latent +λ vert L vert +λ vel L vel .

[0111] In the above formula, λ latent , λvert , λ vel For example, λ is the weight. latent , λ vert , λ vel The value of can be 1.

[0112] Through training in stages 1 and 2, the model is able to effectively learn how to convert audio input into a synchronized 3D facial mesh while maintaining the real-time and high-quality generation process.

[0113] In summary, the embodiments in this specification use an autoregressive diffusion model to select a limited number of past frames as historical facial context and combine them with audio input to generate conditions, guiding the diffusion process to generate facial expression sequences in real time. This overcomes the performance bottleneck of existing methods when generating sequences of arbitrary length and the problem of poor responsiveness to dynamic audio input.

[0114] For example, compared to rule-based methods, the embodiments of this specification can capture the diversity of facial expressions more comprehensively; compared to other learning-based methods, the embodiments of this specification make better use of long-term audio context through an autoregressive mechanism, resulting in more time-stable animations; compared to other diffusion model methods, the autoregressive diffusion model of the embodiments of this specification can flexibly handle sequences of arbitrary length and has advantages in real-time performance.

[0115] The embodiments described in this specification offer significant advantages in terms of generation quality and real-time performance. For example, in some embodiments, a multimodal alignment mechanism makes the generated facial animations visually more realistic and better reflects the semantic information of the audio input. Furthermore, in some embodiments, by employing a conditional distribution of a single-layer MLP approximation inverse process, real-time generation speed can be ensured, meeting the requirements of real-time applications.

[0116] Experimental results show that the proposed method outperforms existing methods on benchmark datasets and can achieve high-quality and real-time audio-driven 3D facial animation generation.

[0117] Therefore, the embodiments described in this specification have broad application prospects in many practical application scenarios, including advertising, virtual reality, game development, and remote communication. It can generate natural and realistic facial expressions, enhancing user immersion and interactive experience, especially in situations with limited network bandwidth, reducing data transmission while maintaining high-quality visual effects.

[0118] This concludes the detailed explanation of the implementation of the audio-driven facial animation generation method according to the embodiments of this specification.

[0119] According to another embodiment, an audio-driven facial animation generation apparatus is provided. Figure 5A schematic diagram of an audio-driven facial animation generation apparatus according to one embodiment is shown. This apparatus can be deployed in any device, platform, or cluster of devices with data storage, computing, and processing capabilities.

[0120] like Figure 5 As shown, the facial animation generation device 500 includes:

[0121] The encoding unit 51 is configured to encode an audio segment into an audio feature sequence composed of audio features from multiple time frames;

[0122] Execution unit 52 is configured to sequentially perform a first prediction operation on any target time frame of the plurality of time frames, including:

[0123] The first feature is obtained by encoding the facial data of h frames prior to the target time frame using the encoder in the autoencoder system.

[0124] The conditional predictor generates first conditional data based on the audio feature sequence, the identity features representing the speaker's identity, and the first feature.

[0125] Using a diffusion model based on the first conditional data, the latent representation is recovered from the noise.

[0126] The latent representation is decoded using the decoder in the autoencoder system to obtain the facial data of the target time frame.

[0127] In some embodiments, when the target time frame belongs to the first h time frames among the plurality of time frames, the facial data of the h frames preceding the target time frame is obtained by initialization.

[0128] In some embodiments, the autoencoder system has a codebook containing primitive vectors of N facial primitives; encoding h frames of facial data prior to a target time frame to obtain a first feature includes: using the encoder to encode the h frames of facial data into a feature matrix consisting of h feature vectors; reshaping the feature matrix into encoding units of multiple facial components; and mapping each encoding unit to the nearest primitive vector in the codebook to obtain the first feature.

[0129] In some embodiments, a conditional predictor generates first conditional data based on the audio feature sequence, the identity feature representing the speaker's identity, and the first feature. This includes: aggregating the first feature and the identity feature; inputting the aggregated first input sequence into a first attention module for self-attention processing to obtain a first output sequence; inputting the first output sequence and the audio feature sequence into a second attention module for cross-modal cross-attention processing to obtain a second output sequence; and inputting the second output sequence into a feedforward neural network for processing to obtain the first conditional data.

[0130] In some further embodiments, the self-attention processing includes: calculating attention scores for each position in the first input sequence, only for the elements of that position and its preceding positions, to obtain a causal attention matrix; and calculating a first output sequence based on the causal attention matrix.

[0131] In some embodiments, recovering a latent representation from noise based on the first conditional data using a diffusion model includes: performing multiple rounds of denoising, each round of denoising including: inputting the current noise, the first conditional data, and round features characterizing the current denoising round into a diffusion head for processing to obtain the latent representation of the current round; performing at least one diffusion process on the latent representation of the current round, and using the diffusion-processed latent representation as the current noise in the next round of denoising.

[0132] In some further embodiments, the diffuser head is a multilayer sensor.

[0133] In some embodiments, the latent representation is decoded using a decoder in the autoencoder system to obtain facial data of the target time frame, including: mapping each feature vector in the latent representation to the nearest primitive vector in the codebook to obtain quantized latent features; and inputting the quantized latent features into the decoder for processing to obtain facial data of the target time frame.

[0134] In some embodiments, the facial animation generation apparatus 300 further includes a generation unit configured to generate a facial animation corresponding to the audio segment based on facial data from the plurality of time frames.

[0135] In some embodiments, facial data includes the position information of V vertices in a facial mesh template.

[0136] In some embodiments, the audio segment is taken from training samples, which further include real facial data corresponding to the audio segment; the facial animation generation apparatus 300 further includes a training unit. The training unit is configured to update the conditional predictor and the diffusion model in the direction of decreasing training loss, wherein the training loss includes latent loss and vertex loss, the latent loss characterizing the difference between the latent representation and the real latent representation corresponding to the real facial data, and the vertex loss characterizing the difference between the predicted vertex states in the generated facial data and the real vertex states in the real facial data.

[0137] In some further embodiments, the vertex loss includes vertex position loss and vertex velocity loss, wherein the vertex position loss represents the distance between the vertex position in the generated face data and the vertex position in the real face data, the vertex velocity loss represents the difference between the generated vertex velocity and the real vertex velocity, the generated vertex velocity represents the distance between vertices in two adjacent frames of generated face data, and the real vertex velocity represents the distance between vertices in two adjacent frames of real face data.

[0138] For specific implementation examples of each unit in the above device, please refer to the previous examples. Figures 1 to 4 The description.

[0139] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0140] According to yet another embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 2 The described method and steps.

[0141] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0142] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0143] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio-driven facial animation generation method, comprising: Encoding an audio segment into an audio feature sequence composed of audio features from multiple time frames; For any target time frame among the plurality of time frames, a first prediction operation is performed sequentially, including: The first feature is obtained by encoding the facial data of h frames prior to the target time frame using the encoder in the autoencoder system. The conditional predictor generates first conditional data based on the audio feature sequence, the identity features representing the speaker's identity, and the first feature. Using a diffusion model based on the first conditional data, the latent representation is recovered from the noise; The latent representation is decoded using the decoder in the autoencoder system to obtain the facial data of the target time frame.

2. The method according to claim 1, wherein, When the target time frame belongs to the first h time frames among the plurality of time frames, the facial data of the h frames before the target time frame is obtained through initialization.

3. The method according to claim 1, wherein, The autoencoder system has a codebook containing primitive vectors of N facial primitives; it encodes facial data from h frames prior to the target time frame to obtain a first feature, including: Using the encoder, the h frames of facial data are encoded into a feature matrix consisting of h feature vectors; The feature matrix is ​​reshaped into encoding units for multiple facial components; The first feature is obtained by mapping each coding unit to the nearest primitive vector in the codebook.

4. The method according to claim 1, wherein, The conditional predictor generates first conditional data based on the audio feature sequence, the identity features representing the speaker's identity, and the first feature, including: The first feature and the identity feature are aggregated, and the first input sequence obtained by aggregation is input into the first attention module for self-attention processing to obtain the first output sequence; The first output sequence and the audio feature sequence are input together into the second attention module for cross-modal cross-attention processing to obtain the second output sequence; The second output sequence is input into a feedforward neural network for processing to obtain the first conditional data.

5. The method according to claim 4, wherein, The self-attention processing includes: For each position in the first input sequence, attention scores are calculated only for the elements of that position and the positions preceding it, resulting in a causal attention matrix; The first output sequence is calculated based on the causal attention matrix.

6. The method according to claim 1, wherein, Based on the first conditional data, a latent representation is recovered from the noise using a diffusion model, including: performing multiple rounds of denoising, each round of denoising including: The current noise, the first conditional data, and the round features representing the current denoising round are input into the diffusion head for processing to obtain the potential representation of the current round. The latent representation of the current round is subjected to at least one diffusion process, and the diffused latent representation is used as the current noise in the next round of denoising.

7. The method according to claim 6, wherein, The diffuser head is a multilayer sensor.

8. The method according to claim 3, wherein, The latent representation is decoded using the decoder in the autoencoder system to obtain the facial data of the target time frame, including: Each feature vector in the latent representation is mapped to the nearest primitive vector in the codebook to obtain the quantized latent features; The quantized latent features are input into the decoder for processing to obtain the facial data of the target time frame.

9. The method according to claim 1, wherein, The method further includes: generating a facial animation corresponding to the audio segment based on the facial data of the plurality of time frames.

10. The method according to claim 1, wherein, The facial data includes the position information of V vertices in the facial mesh template.

11. The method according to claim 10, wherein, The audio segment is taken from the training samples, which also include real facial data corresponding to the audio segment; the method further includes: In the direction of reducing training loss, the conditional predictor and the diffusion model are updated. The training loss includes latent loss and vertex loss. The latent loss characterizes the difference between the latent representation and the real latent representation corresponding to the real face data. The vertex loss characterizes the difference between the predicted vertex state in the generated face data and the real vertex state in the real face data.

12. The method according to claim 11, wherein, The vertex loss includes vertex position loss and vertex velocity loss. The vertex position loss represents the distance between the vertex position in the generated face data and the vertex position in the real face data. The vertex velocity loss represents the difference between the generated vertex velocity and the real vertex velocity. The generated vertex velocity represents the distance between vertices in two adjacent frames of generated face data. The real vertex velocity represents the distance between vertices in two adjacent frames of real face data.

13. An audio-driven facial animation generation device, comprising: The encoding unit is configured to encode an audio segment into an audio feature sequence consisting of audio features from multiple time frames; The execution unit is configured to sequentially perform a first prediction operation on any target time frame of the plurality of time frames, including: The first feature is obtained by encoding the facial data of h frames prior to the target time frame using the encoder in the autoencoder system. The conditional predictor generates first conditional data based on the audio feature sequence, the identity features representing the speaker's identity, and the first feature. Using a diffusion model based on the first conditional data, the latent representation is recovered from the noise; The latent representation is decoded using the decoder in the autoencoder system to obtain the facial data of the target time frame.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-12.

15. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-12.

Images