Digital human expression generation method based on multi-modal feature fusion and emotion enhancement

Abstract

The application discloses a digital human expression generation method based on multi-modal feature fusion and emotion enhancement, and belongs to the technical field of generative artificial intelligence, comprising the following steps: extracting speech semantic features and speech emotion features based on driving speech, extracting text emotion features based on emotional text, and extracting visual latent variables and image semantic features based on reference images; fusing the speech emotion features and the text emotion features through a perception resampling mechanism to generate fusion emotion control features; inputting the fusion emotion control features, the speech semantic features, the visual latent variables and the image semantic features into a DiT video generation model as conditional inputs to generate a denoising video latent vector, which is mapped into a facial expression latent vector by a Transformer adapter, and then restored into a facial expression parameter sequence by a FaceVerse decoder to drive a digital human.The application can realize end-to-end generation of high-fidelity and emotion-controllable facial expression parameters from speech and text.

Description

Technical Field

[0001] This invention belongs to the field of generative artificial intelligence technology, specifically relating to a method for generating digital human facial expressions based on multimodal feature fusion and emotion enhancement. Background Technology

[0002] In recent years, with the rapid development of AI-generated content (AIGC) technology, video generation, as one of the most challenging and promising areas, has received widespread attention in scenarios such as digital humans, virtual anchors, film and television post-production, games, and immersive interactions. Large-scale pre-trained video generation models, such as Sora, WAN, and StableAvatar, supported by massive amounts of data and Diffused Transformer (DiT) backbone networks, are now capable of generating high-quality videos with good spatiotemporal consistency, rich detail, and considerable duration, providing powerful general priors for multimodal generation tasks.

[0003] Among numerous applications, voice-driven facial expression generation is particularly crucial. This task uses voice as the primary driving signal to synthesize animated facial videos that are highly consistent with the voice content, emotion, and speaker style based on the input speech. From an application perspective, voice-driven facial expression generation technology can significantly lower the barrier to content production, enhance immersion and emotional expression, and is expected to replace some expensive motion capture equipment and manual binding processes, thus significantly improving production efficiency.

[0004] Currently, most traditional speech-driven facial expression generation methods design network structures and loss functions independently based on the task, typically learning the mapping from speech to 3DMM parameters or vertex displacements directly on relatively small-scale 3D datasets. Representative methods include VOCA based on convolutional networks, FaceFormer based on Transformer, CodeTalker based on VQ-VAE discrete motion representation, and FaceDiffuser based on diffusion models. These methods have certain advantages in terms of control accuracy and task adaptability.

[0005] However, despite the progress made in existing technologies, a series of key technical challenges still exist in practical applications, mainly in the following three aspects: First, a technological gap exists between existing speech-driven facial expression generation models and advanced pre-trained video models. Traditional methods primarily rely on limited-scale 3D datasets for training, failing to fully utilize the rich motion priors and visual knowledge inherent in large-scale pre-trained video models. On the other hand, general-purpose pre-trained video models (such as DiT-based models) are mostly text- or image-driven. While some works have attempted to introduce audio-driven features, they generally employ simple audio encoder injection schemes, failing to explicitly address the distribution drift problem caused by the lack of audio priors in the diffusion backbone. Currently, a mature solution is lacking for systematically integrating speech-driven tasks with pre-trained video models to achieve both strong generalization capabilities and fine-grained controllability.

[0006] Secondly, there is a lack of emotional expression. Existing works on speech-driven facial expression generation primarily focus on lip-sync and speech content consistency, with relatively weak modeling and controllable generation of emotional information in speech. Most existing methods modulate the model through explicit emotion labels or discrete emotion categories. This approach is highly dependent on discrete emotion annotation and struggles to capture complex and subtle emotional changes (such as prosody, stress, and intonation). While some methods attempt to implicitly extract emotional information from audio, the generated expressions often appear flat and lack emotional impact without an effective decoupling mechanism.

[0007] Finally, high-quality multimodal datasets are lacking. Speech-driven facial expression generation highly depends on paired speech, video, and facial expression parameter data. Existing public datasets (such as BIWI, VOCASET, 3DMEAD, etc.) are insufficient in terms of scale, number of identities, and emotional diversity, especially lacking high-quality data with simultaneous annotation of "speech-emotion-facial expression parameters," which directly limits the training effect and generalization ability of models at large scales.

[0008] Therefore, how to integrate the powerful priors of pre-trained video models, achieve nuanced and controllable emotional expression, and overcome the bottleneck of scarce high-quality multimodal data has become a key challenge to promote the practical application and high performance of voice-driven facial expression generation technology. Summary of the Invention

[0009] In view of the above, the purpose of this invention is to provide a digital human facial expression generation method based on multimodal feature fusion and emotion enhancement. This method utilizes multimodal features extracted from driving speech and emotional text for deep fusion to obtain emotion control features. Furthermore, it maps denoised video latent vectors generated by a pre-trained video generation model to facial expression latent vectors via an adapter. These vectors serve as driving conditions to generate high-fidelity digital human facial animations that are highly consistent with the prosody and semantics of the input speech and text, exhibiting nuanced emotional expression. This effectively solves the cross-modal alignment problem between general video generation models and facial geometric representations. A phased training strategy is constructed, and fine-grained emotion condition control is introduced, improving the generalization ability of the generation model under different identities and complex emotional scenarios. This method has broad application prospects in multiple fields such as virtual anchors, film and television production, and human-computer interaction.

[0010] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a digital human expression generation method based on multimodal feature fusion and emotion enhancement, comprising the following steps: Speech semantic features and speech emotion features are extracted based on driving speech, text emotion features are extracted based on emotional text, and visual latent variables and image semantic features are extracted based on reference images, respectively. Voice emotion features and text emotion features are fused through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features; The DiT video generation model, which integrates emotion control features, speech semantic features, visual latent variables and image semantic features as conditional inputs, generates temporally coherent denoised video latent vectors through a stream matching denoising process. A pre-trained Transformer adapter is used to map the latent vectors of denoised video to latent vectors of facial expressions, and a pre-trained FaceVerse decoder is used to restore the latent vectors of facial expressions to a sequence of facial expression parameters to drive the digital human to generate emotion-enhanced facial animations.

[0011] Preferably, the step of fusing speech emotion features and text emotion features through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features includes: Initialize a fixed number of learnable query vectors and concatenate speech sentiment features with text sentiment features into a multimodal sentiment feature sequence; Learnable query vectors are used as queries in the cross-attention mechanism, and multimodal sentiment feature sequences are used as keys and values ​​to interact and aggregate, outputting fixed-dimensional fused sentiment control features.

[0012] Preferably, the DiT video generation model, which is pre-trained using emotion control features, speech semantic features, visual latent variables, and image semantic features as conditional inputs, generates temporally coherent denoised video latent vectors through a stream matching denoising process, including: A DiT video generation model is constructed based on the diffusion Transformer architecture, which contains multiple diffusion Transformer blocks. Each diffusion Transformer block contains a self-attention layer, an emotion cross-attention layer, and an image-speech cross-attention layer. In each denoising iteration, the noise latent vector obtained by adding noise to the visual latent variables is input into the attention layer to capture the spatiotemporal dependencies within the video latent sequence; the fused emotion control features are input into the emotion cross-attention layer to inject global emotion style into the generation process; and the speech semantic features and image semantic features are input into the image-speech cross-attention layer to ensure that lip movements are synchronized with pronunciation and to maintain identity consistency. The velocity field from the noise distribution to the potential distribution of the target video is predicted iteratively based on the above process using the flow matching algorithm, and finally the temporally coherent denoised video potential vector is generated.

[0013] Preferably, the step of mapping the denoised video latent vectors to facial expression latent vectors using a pre-trained Transformer adapter includes: A Transformer adapter is built based on the Transformer architecture, which includes a spatiotemporal alignment layer, a linear mapping layer, a one-dimensional convolutional layer, and a multi-layer self-attention module. The latent vector of the denoised video is input into the spatiotemporal alignment layer to eliminate the spatial dimension of the features and convert them into one-dimensional temporal features. The one-dimensional temporal features are then input into the linear mapping layer for projection and smoothed by the one-dimensional convolutional layer. The smoothed features are then input into the multi-layer self-attention module for long-range temporal modeling and nonlinear mapping, and finally output as a facial expression latent vector aligned with the expression latent space.

[0014] Preferably, the step of reconstructing the latent facial expression vector into a sequence of facial expression parameters using a pre-trained FaceVerse decoder includes: The FaceVerse codec is built based on the VQ-VAE architecture, and the FaceVerse decoder containing the quantization codebook and the symmetric temporal upsampling module is extracted for inference. The FaceVerse decoder converts the latent vectors of facial expressions into discrete embedding sequences by querying the quantization codebook, and then reconstructs the discrete embedding sequences into a sequence of facial expression parameters at the original frame rate through a temporal upsampling module.

[0015] Preferably, the DiT video generation model, Transformer adapter, and FaceVerse decoder are optimized using a three-stage training strategy, including: The first stage is FaceVerse codec pre-training: the FaceVerse codec is self-supervised and trained using facial expression parameter data, the geometric parameter reconstruction loss and quantization loss are optimized, and the pre-trained FaceVerse decoder part is extracted for use in the inference stage. The second stage is cross-modal adapter training: freeze the pre-trained DiT video generation model and the FaceVerse decoder trained in the first stage, and train the Transformer adapter using paired video-expression parameter data to optimize the geometric parameter reconstruction loss and latent space mapping loss. In the third stage, joint fine-tuning of emotion enhancement is performed: the FaceVerse decoder trained in the first stage is frozen, and the emotion cross-attention layer in the DiT video generation model, the Transformer adapter trained in the second stage, and the emotion fusion module are jointly trained using multimodal data with emotion annotations to optimize the stream matching loss and geometric parameter reconstruction loss.

[0016] Secondly, embodiments of the present invention also provide a digital human expression generation device based on multimodal feature fusion and emotion enhancement, which is implemented using the above-mentioned digital human expression generation method based on multimodal feature fusion and emotion enhancement, including: a multimodal feature extraction module, an emotion feature fusion module, a video latent feature generation module, and a cross-modal adaptation decoding module; The multimodal feature extraction module is used to extract speech semantic features and speech emotion features based on driving speech, extract text emotion features based on emotional text, and extract visual latent variables and image semantic features based on reference images, respectively. The emotional feature fusion module is used to fuse speech emotional features and text emotional features through an emotional fusion module based on a perceptual resampling mechanism to generate fused emotional control features; The video latent feature generation module is used to take the DiT video generation model pre-trained with emotion control features, speech semantic features, visual latent variables and image semantic features as conditional inputs, and generate temporally coherent denoised video latent vectors through a stream matching denoising process. The cross-modal adaptation decoding module is used to map the denoised video latent vectors to facial expression latent vectors using a pre-trained Transformer adapter, and to restore the facial expression latent vectors to a sequence of facial expression parameters using a pre-trained FaceVerse decoder, so as to drive the digital human to generate emotion-enhanced facial animations.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described digital human expression generation method based on multimodal feature fusion and emotion enhancement when executing the computer program.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described digital human expression generation method based on multimodal feature fusion and emotion enhancement.

[0019] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) By constructing and fine-tuning the pre-trained DiT video generation model, this invention makes full use of the general motion and visual priors learned in large-scale data, and achieves a significant improvement in the temporal coherence, naturalness and generalization ability of generated expressions.

[0020] (2) This invention designs a multimodal emotion fusion framework based on a perceptual resampling mechanism, and bridges the visual and geometric spaces through a Transformer adapter. Finally, based on the FaceVerse decoder, it realizes fine-grained and continuously controllable expression of emotional style from semantics to rhythm, which effectively enhances the appeal and expressiveness of the generated expressions.

[0021] (3) This invention optimizes geometric representation, cross-modal mapping, and emotion control capabilities sequentially through a phased, progressive training paradigm. First, a high-quality geometric prior is constructed through pre-training in the first stage. Then, the mapping from video to geometry is learned through training in the second stage, reducing dependence on paired data. Finally, through fine-tuning in the third stage, the transfer and enhancement of emotion control capabilities are achieved. This enables the generation of high-fidelity, emotionally nuanced, and lip-synced digital human facial animations. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the digital human facial expression generation method based on multimodal feature fusion and emotion enhancement provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall architecture of the digital human expression generation method based on multimodal feature fusion and emotion enhancement provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the feature fusion module architecture based on the perceptual resampling mechanism provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the architecture of the Transformer adapter provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the architecture of the FaceVerse codec based on VQ-VAE provided in an embodiment of the present invention; Figure 6 This is a visual rendering diagram of the FaceVerse 3D facial model provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the digital human expression generation device based on multimodal feature fusion and emotion enhancement provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] The inventive concept of this invention is as follows: Addressing the bottlenecks in existing technologies, such as the disconnect between pre-trained large video models and speech-driven tasks, insufficient emotional expression capabilities, and reliance on high-quality paired data, this invention provides a digital human facial expression generation method based on multimodal feature fusion and emotional enhancement. This method deeply fuses speech and text emotional features using a perceptual resampling mechanism, and generates video latent vectors using a DiT video generation model with multimodal fusion features as conditions. A Transformer adapter is then designed to achieve cross-modal mapping from video features to geometric parameters. Finally, a FaceVerse decoder restores the mapped facial expression latent vectors to a sequence of facial expression parameters. Combined with a three-stage training process, this ultimately achieves end-to-end generation of high-fidelity, emotionally enhanced facial expression parameters from speech and text.

[0026] Figure 1 This is a flowchart illustrating the digital human expression generation method based on multimodal feature fusion and emotion enhancement provided in this embodiment of the invention, showing the four core steps from data input to the final digital human driving process. Figure 2 This is a schematic diagram of the overall architecture of the digital human expression generation method based on multimodal feature fusion and emotion enhancement provided in the embodiments of the present invention. It shows in detail the internal module connection relationship of the multimodal feature extraction and emotion enhancement stage, the cross-modal facial latent feature generation stage, and the facial representation mapping and parameter decoding stage.

[0027] like Figure 1 and Figure 2 As shown in the embodiment, a digital human expression generation method based on multimodal feature fusion and emotion enhancement is provided, including the following steps: S1 extracts speech semantic features and speech emotion features based on driving speech, extracts text emotion features based on emotional text, and extracts visual latent variables and image semantic features based on reference images, respectively.

[0028] In this embodiment, to achieve fine-grained emotion control, the following steps are taken: first, the driving speech (specifically, a segment of driving speech audio), the emotional text (specifically, a segment of emotional description text), and the reference image (specifically, a reference face image) are acquired. The Wav2Vec 2.0 speech content encoder is used to extract the speech semantic features of the driving speech. These features mainly contain speech semantic information to ensure accurate synchronization between lip movements and the spoken content. The Emotion2Vec speech emotion encoder is used to extract the speech emotion features of the driving speech (including deep emotion features such as intonation and emotional intensity). Emotion2Vec has undergone self-supervised pre-training on massive amounts of unlabeled emotion data, enabling it to keenly capture paralinguistic information such as intonation, loudness, and speech rate, and extract sentence-level emotion features. The UMT5 text encoder is used to extract the text emotion features of the emotional text, providing global emotional style guidance. The 3D-VAE visual encoder is used to extract the visual latent variables of the reference image, which are used to construct the latent space training target for the video generation model. The CLIP image encoder is used to extract the image semantic features of the reference image (including identity information) to ensure the consistency of the generated results.

[0029] S2, the speech emotion features and text emotion features are fused through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features.

[0030] like Figure 3The diagram illustrates the feature fusion module architecture based on a perceptive resampling mechanism, demonstrating how learnable query vectors are used to interact and compress speech and text sentiment features. In this embodiment, to address the misalignment in length and semantic space between speech features (sentence-level) and text features (sequence-level), a sentiment fusion module based on a perceptive resampling mechanism is introduced. This module fuses implicit speech sentiment features with explicit emotional descriptions in the text into a unified sentiment control feature, which serves as a conditional input to the forward inference of the generative model. This allows the model to simultaneously satisfy both speech sentiment cues and target sentiment constraints when generating facial expressions, thereby enhancing and stabilizing control over expression amplitude, emotional tendency, and dynamic rhythm. Specifically, a fixed number of learnable query vectors are initialized, and the speech sentiment features (processed into speech sentiment vectors via a linear layer) and text sentiment features are concatenated along the sequence dimension to obtain a multimodal sentiment feature sequence. Next, a cross-attention mechanism is used, with a learnable query vector as the query and a multimodal sentiment feature sequence as the key and value. This interaction and aggregation process results in the output sentiment features via a feedforward layer. In this way, the query vector automatically retrieves and aggregates the most crucial sentiment cues from variable-length multimodal inputs. The sentiment features are then concatenated with the original text sentiment features to obtain concatenated sentiment features. Finally, a padding layer unifies the feature dimensions according to actual needs, resulting in a final fixed-dimensional, semantically and prosodic fused sentiment control feature.

[0031] S3 uses a DiT video generation model pre-trained with emotion control features, speech semantic features, visual latent variables and image semantic features as conditional inputs to generate temporally coherent denoised video latent vectors through a stream matching denoising process.

[0032] In this embodiment, a video generation model based on the Diffusion Transformer (DiT) architecture is used as the backbone. The original text cross-attention layer is transformed into an emotion cross-attention layer, specifically designed to receive the aforementioned fused emotion control features. The DiT video generation model, built on the Diffusion Transformer architecture, comprises multiple Diffusion Transformer blocks. Each Diffusion Transformer block includes a self-attention layer, an emotion cross-attention layer, and an image-speech cross-attention layer, as shown below. Figure 2 As shown.

[0033] In each denoising iteration, the noise latent vector obtained by adding noise to the visual latent variables is input into the attention layer to capture the spatiotemporal dependencies within the video latent sequence; fused emotion control features are input into the emotion cross-attention layer to inject global emotional style into the generation process; and speech semantic features and image semantic features are input into the image-speech cross-attention layer to ensure lip-sync with pronunciation and maintain identity consistency. Under multiple guidance, the denoising process is executed through the FlowMatching algorithm, and the velocity field from the noise distribution to the target video latent distribution is predicted iteratively based on the above process. Compared with the traditional diffusion model, the FlowMatching mechanism can generate high-quality, temporally coherent denoised video latent vectors with fewer sampling steps.

[0034] S4 utilizes a pre-trained Transformer adapter to map the latent vectors of the denoised video to latent vectors of facial expressions, and then uses a pre-trained FaceVerse decoder to restore the latent vectors of facial expressions to a sequence of facial expression parameters, thereby driving the digital human to generate emotion-enhanced facial animations.

[0035] S4.1 is a lightweight adapter built on the Transformer architecture. It includes a spatiotemporal alignment layer, a linear mapping layer, a one-dimensional convolutional layer, and a multi-layer self-attention module, serving as a bridge connecting the visual modality and the geometric modality.

[0036] like Figure 4 The diagram illustrates the architecture of the Transformer adapter, showcasing the processing flow of video latent features through the spatiotemporal alignment module, linear mapping layer, one-dimensional convolutional layer, and Transformer Block. In this embodiment, addressing the mismatch between the general visual features output by the large video generation model and facial geometric features, this invention constructs a Transformer adapter to receive the denoised video latent vector output by the upstream DiT video generation model. First, the denoised video latent vector enters the spatiotemporal alignment layer. Since this vector typically contains spatial dimensions (height × width) and temporal dimensions, the spatial dimension is eliminated through the adaptive average pooling layer and flattening operation of the spatiotemporal alignment layer, normalizing the three-dimensional tensor into a unified one-dimensional temporal feature sequence, thus adapting it to the sequence characteristics of the geometric parameters. Subsequently, the one-dimensional temporal features are projected into the linear mapping layer and then smoothed by the one-dimensional convolutional layer, enhancing local temporal smoothness and compensating for the Transformer's inability to capture fine-grained local features. The smoothed features are then input into the backbone network, which consists of multiple Transformer Blocks. The network uses multi-layer self-attention modules for long-term temporal modeling and feedforward nonlinear mapping to project the semantic space of the video features nonlinearly into the latent space distribution of the FaceVerse geometric model. Finally, the output is a facial expression latent vector aligned with the expression latent space (i.e., the VQ-VAE codebook space).

[0037] S4.2 is based on the VQ-VAE architecture to build the FaceVerse codec, which includes a temporal downsampling module (encoder part) and a temporal upsampling module (decoder part) for the discretization and reconstruction of facial parameters.

[0038] like Figure 5 The diagram illustrates the architecture of the FaceVerse encoder / decoder based on VQ-VAE, showcasing the encoding process from facial expression parameter sequences to discrete latent vectors and the decoding and reconstruction process from latent vectors to parameter sequences. In this embodiment, to address the high-dimensional redundancy and temporal jitter issues of facial expression parameters, this invention does not directly generate them in the original parameter space. Instead, it first constructs a compact, discrete latent space with manifold properties. The encoder receives high-dimensional, high-frame-rate (e.g., 25fps) facial expression parameters (including expression coefficients and head pose coefficients). The encoder integrates a linear projection layer and a temporal downsampling module. Using a cascade of causal one-dimensional convolutions with a stride of 2, the long sequence of facial parameters is compressed in the temporal dimension (e.g., compressed to 1 / 4 of the original length), resulting in low-dimensional, low-frame-rate latent features. Simultaneously, six Transformer layers are used to extract high-dimensional features, thereby expanding the model's temporal receptive field and capturing macroscopic patterns of expression evolution. Furthermore, a learnable discrete codebook is introduced, mapping the continuous latent features output by the encoder to the nearest-neighbor discrete embedding vectors in the codebook. This process forces the model to learn the "primitives" of facial movements (similar to the concept of phonemes) rather than memorizing specific parameter values, thus effectively avoiding oversmoothing in the generation process and enhancing the robustness of the movements.

[0039] Then, using a decoder symmetrical to the encoder, a "channel-to-time" rearrangement strategy (such as a one-dimensional variant of PixelShuffle) is employed. This is followed by 6 Transformer layers, 2 causal one-dimensional convolutional layers (followed by channel-time rearrangement layers), and a fully connected layer. This process losslessly restores the quantized discrete latent features to a high-fidelity facial expression parameter sequence at the original frame rate, including 177-dimensional FaceVerse expression coefficients and 3-dimensional head pose coefficients. During model inference, only the FaceVerse decoder containing the quantized codebook and the symmetrical temporal upsampling module is extracted and applied.

[0040] Furthermore, the DiT video generation model, Transformer adapter, and FaceVerse decoder are optimized through a three-stage training strategy.

[0041] The first stage is FaceVerse codec pre-training: The FaceVerse codec is self-supervised and trained using facial expression parameter data (including 177-dimensional FaceVerse expression coefficients and 3-dimensional head pose coefficients). The geometric parameter reconstruction loss (including expression coefficient loss, head rotation coefficient loss and 3D vertex position loss) and quantization loss (Codebook Loss) are optimized to provide a stable geometric representation basis for subsequent generation tasks. The pre-trained FaceVerse decoder part is extracted for use in the inference stage.

[0042] The total loss function used in the first stage of pre-training The definition is as follows: , in, Represents the geometric parameter reconstruction loss. Indicates quantified loss. and These represent the weights of the geometric parameter reconstruction loss and the quantization loss, respectively.

[0043] Geometric parameter reconstruction loss Represented as: , in, This indicates the loss of the expression coefficient. This indicates the loss of head rotation coefficient. This represents the loss of 3D vertex position. , and The weights of the three types of losses are represented in that order.

[0044] Indicates expression coefficient loss and head rotation coefficient loss The regression accuracy of the parameter space is directly constrained using L1 loss (MAE), expressed as: , , in, The coefficient represents the true expression. To predict the expression coefficient, This represents the actual head rotation coefficient. To predict the head rotation coefficient, Index for iteration count, This represents the total number of iterations.

[0045] 3D vertex position loss Explicit vertex supervision is established using L1 loss (MAE) to ensure the physical consistency of the generated parameters in 3D geometric space. The specific calculation process is as follows: First, using FaceVerse's deformation model dictionary (containing average shape...) And linear expression blendshape bases The expression offset is calculated and the base vertices are reconstructed through matrix multiplication. , Indicates the first The corresponding coefficients of the linear table case of basis variation are used to control the contribution weight of the basis vector in the final face geometry; This represents the first expression in the FaceVerse subspace. A linear facial deformation basis is used, which describes a set of normalized geometric displacement patterns in three-dimensional vertex space. Subsequently, the predicted head rotation Euler angles are calculated. Convert to rotation matrix This is then applied to the base vertices to obtain the final predicted vertex positions. Finally, 478 index points corresponding to MediaPipe keypoints in the FaceVerse topology were selected, and the positions of these keypoints in the prediction grid were calculated. With real grid The L1 distance between them is used as the loss value: , in, This indicates the prediction of 3D vertex coordinates. This represents the actual 3D vertex coordinates, and the superscript idx indicates the 478 index points corresponding to the MediaPipe keypoints in the FaceVerse topology.

[0046] Quantification of loss The codebook update loss and commitment loss, adopted from the VQ-VAE standard, are used to constrain the encoder output approximation codebook vector, and are expressed as follows: , in, This indicates that the loss term has been updated. Indicates the promised loss item, This represents the stop-gradient operator. This represents the latent vector output by the encoder. This represents the latent vector input to the decoder after quantization; The weights representing the commitment loss are used to constrain the encoder output vector. It moves closer to its corresponding quantized codebook vector, thereby avoiding excessive drift in the encoder output.

[0047] The second stage involves cross-modal adapter training: The pre-trained DiT video generation model and the FaceVerse decoder trained in the first stage are frozen. Using paired video-expression parameter data, the Transformer adapter is trained under supervision, with the features output by the DiT video generation model as input and the true latent vectors generated by the FaceVerse encoder as the target. By minimizing the geometric parameter reconstruction loss and latent space mapping loss, a transformation pathway from visual perception to geometry-driven processing is established.

[0048] The total loss function used in the second stage of pre-training The definition is as follows: , in, The geometric parameter reconstruction loss after being restored by the FaceVerse decoder is defined as in the first stage. For latent space mapping loss, and These are the weights for the two types of losses, respectively.

[0049] Latent space mapping loss Defined as the predicted latent vector of the adapter output Compared with the real FaceVerse latent vector Mean squared error (MSE) between: , in, Indicates the first The predicted latent vector output by the next iteration adapter. Indicates and The corresponding FaceVerse encoder generates the true latent vector. .

[0050] The third stage involves joint fine-tuning of emotion enhancement: freezing other feature extractors and the FaceVerse decoder trained in the first stage, and jointly training the emotion cross-attention layer, the Transformer adapter trained in the second stage, and the emotion fusion module in the DiT video generation model using multimodal data with emotion annotations (automatically annotated and cleaned through a large model). Optimization objectives include stream matching loss (optimizing the generated trajectory) and geometric parameter reconstruction loss (ensuring the accuracy of the final output 3D parameters), giving the model the ability to generate subtle micro-expressions based on speech prosody and text commands.

[0051] The total loss function used in the third-stage joint fine-tuning The definition is as follows: , in, For flow matching loss; The geometric parameter reconstruction loss is used to constrain the accuracy of the final generated facial expression parameters, and is defined in the same two-stage manner. Reconstruct loss weights for geometric parameters.

[0052] Stream matching loss Used to supervise the DiT video generation model's prediction from noise distribution To data distribution velocity field The calculation formula is: , in, The speed of prediction for the DiT video generation model. For time steps State interpolation, and They are respectively fused emotion control features, speech semantic features, and image semantic features.

[0053] like Figure 6 The image shown is a visualization rendering of a FaceVerse 3D facial model. The left side of the image displays the skin texture. Figure 6 (a) of the light and shadow ( Figure 6 The realistic rendering effect of (b) in the middle, and the two pictures on the right ( Figure 6 (c) and (d) in the figure show the corresponding three-dimensional geometric mesh structure. While maintaining the consistency of the character's identity, it shows lip movements synchronized with the speech, as well as micro-expressions and head movements that conform to the emotional context.

[0054] Based on the same inventive concept, such as Figure 7 As shown, this embodiment of the invention also provides a digital human expression generation device 700 based on multimodal feature fusion and emotion enhancement, including: a multimodal feature extraction module 710, an emotion feature fusion module 720, a video latent feature generation module 730, and a cross-modal adaptation decoding module 740.

[0055] The multimodal feature extraction module 710 is used to extract speech semantic features and speech emotion features based on driving speech, extract text emotion features based on emotional text, and extract visual latent variables and image semantic features based on reference images, respectively.

[0056] The emotion feature fusion module 720 is used to fuse speech emotion features and text emotion features through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features.

[0057] The video latent feature generation module 730 is used to generate temporally coherent denoised video latent vectors by taking the DiT video generation model pre-trained with emotion control features, speech semantic features, visual latent variables and image semantic features as conditional inputs and using the stream matching denoising process.

[0058] The cross-modal adaptation decoding module 740 is used to map the denoised video latent vectors to facial expression latent vectors using a pre-trained Transformer adapter, and to restore the facial expression latent vectors to a sequence of facial expression parameters using a pre-trained FaceVerse decoder, so as to drive the digital human to generate emotion-enhanced facial animations.

[0059] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described digital human expression generation method based on multimodal feature fusion and emotion enhancement when executing the computer program.

[0060] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described digital human expression generation method based on multimodal feature fusion and emotion enhancement.

[0061] It should be noted that the digital human expression generation device, electronic device, and computer-readable storage medium based on multimodal feature fusion and emotion enhancement provided in the above embodiments all belong to the same inventive concept as the digital human expression generation method based on multimodal feature fusion and emotion enhancement. For details of its implementation process, please refer to the embodiments of the digital human expression generation method based on multimodal feature fusion and emotion enhancement, which will not be repeated here.

[0062] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating digital human facial expressions based on multimodal feature fusion and emotion enhancement, characterized in that, Includes the following steps: Speech semantic features and speech emotion features are extracted based on driving speech, text emotion features are extracted based on emotional text, and visual latent variables and image semantic features are extracted based on reference images, respectively. Voice emotion features and text emotion features are fused through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features; A DiT video generation model with multiple DiT Transformer blocks is constructed based on the DiT Transformer architecture. Each DiT Transformer block contains a self-attention layer, an emotion cross-attention layer, and an image-speech cross-attention layer. The DiT video generation model, which integrates emotion control features, speech semantic features, visual latent variables, and image semantic features as conditional inputs, generates temporally coherent denoised video latent vectors through a stream matching denoising process. Based on the Transformer architecture, a Transformer adapter is constructed, which includes a spatiotemporal alignment layer, a linear mapping layer, a one-dimensional convolutional layer, and a multi-layer self-attention module. The pre-trained Transformer adapter is used to map the latent vectors of denoised video into latent vectors of facial expressions. Based on the VQ-VAE architecture, a FaceVerse codec is constructed, and a FaceVerse decoder containing a quantization codebook and a symmetric temporal upsampling module is extracted. The pre-trained FaceVerse decoder is used to restore the latent vectors of facial expressions into a sequence of facial expression parameters to drive the digital human to generate emotion-enhanced facial animations. The DiT video generation model, Transformer adapter, and FaceVerse decoder are optimized through a three-stage training strategy, including: The first stage is FaceVerse codec pre-training: the FaceVerse codec is self-supervised and trained using facial expression parameter data, the geometric parameter reconstruction loss and quantization loss are optimized, and the pre-trained FaceVerse decoder part is extracted for use in the inference stage. The second stage is cross-modal adapter training: freeze the pre-trained DiT video generation model and the FaceVerse decoder trained in the first stage, and train the Transformer adapter using paired video-expression parameter data to optimize the geometric parameter reconstruction loss and latent space mapping loss. In the third stage, joint fine-tuning of emotion enhancement is performed: the FaceVerse decoder trained in the first stage is frozen, and the emotion cross-attention layer in the DiT video generation model, the Transformer adapter trained in the second stage, and the emotion fusion module are jointly trained using multimodal data with emotion annotations to optimize the stream matching loss and geometric parameter reconstruction loss.

2. The digital human expression generation method based on multimodal feature fusion and emotion enhancement according to claim 1, characterized in that, The process of fusing speech emotion features and text emotion features through an emotion fusion module based on a perceptual resampling mechanism to generate fused emotion control features includes: Initialize a fixed number of learnable query vectors and concatenate speech sentiment features with text sentiment features into a multimodal sentiment feature sequence; Learnable query vectors are used as queries in the cross-attention mechanism, and multimodal sentiment feature sequences are used as keys and values ​​to interact and aggregate, outputting fixed-dimensional fused sentiment control features.

3. The digital human expression generation method based on multimodal feature fusion and emotion enhancement according to claim 1, characterized in that, The DiT video generation model, which is pre-trained by fusing emotion control features, speech semantic features, visual latent variables, and image semantic features as conditional inputs, generates temporally coherent denoised video latent vectors through a stream matching denoising process, including: In each denoising iteration, the noise latent vector obtained by adding noise to the visual latent variables is input into the attention layer to capture the spatiotemporal dependencies within the video latent sequence; the fused emotion control features are input into the emotion cross-attention layer to inject global emotion style into the generation process; and the speech semantic features and image semantic features are input into the image-speech cross-attention layer to ensure that lip movements are synchronized with pronunciation and to maintain identity consistency. The velocity field from the noise distribution to the potential distribution of the target video is predicted iteratively based on the above process using the flow matching algorithm, and finally the temporally coherent denoised video potential vector is generated.

4. The digital human expression generation method based on multimodal feature fusion and emotion enhancement according to claim 1, characterized in that, The process of mapping denoised video latent vectors to facial expression latent vectors using a pre-trained Transformer adapter includes: The latent vector of the denoised video is input into the spatiotemporal alignment layer to eliminate the spatial dimension of the features and convert them into one-dimensional temporal features. The one-dimensional temporal features are then input into the linear mapping layer for projection and smoothed by the one-dimensional convolutional layer. The smoothed features are then input into the multi-layer self-attention module for long-range temporal modeling and nonlinear mapping, and finally output as a facial expression latent vector aligned with the expression latent space.

5. The digital human expression generation method based on multimodal feature fusion and emotion enhancement according to claim 1, characterized in that, The process of reconstructing the latent facial expression vector into a sequence of facial expression parameters using a pre-trained FaceVerse decoder includes: The FaceVerse decoder converts the latent vectors of facial expressions into discrete embedding sequences by querying the quantization codebook, and then reconstructs the discrete embedding sequences into a sequence of facial expression parameters at the original frame rate through a temporal upsampling module.

6. A digital human expression generation device based on multimodal feature fusion and emotion enhancement, implemented using the digital human expression generation method based on multimodal feature fusion and emotion enhancement as described in any one of claims 1 to 5, characterized in that, include: Multimodal feature extraction module, emotion feature fusion module, video latent feature generation module, cross-modal adaptation decoding module; The multimodal feature extraction module is used to extract speech semantic features and speech emotion features based on driving speech, extract text emotion features based on emotional text, and extract visual latent variables and image semantic features based on reference images, respectively. The emotional feature fusion module is used to fuse speech emotional features and text emotional features through an emotional fusion module based on a perceptual resampling mechanism to generate fused emotional control features; The video latent feature generation module is used to take the DiT video generation model pre-trained with emotion control features, speech semantic features, visual latent variables and image semantic features as conditional inputs, and generate temporally coherent denoised video latent vectors through a stream matching denoising process. The cross-modal adaptation decoding module is used to map the denoised video latent vectors to facial expression latent vectors using a pre-trained Transformer adapter, and to restore the facial expression latent vectors to a sequence of facial expression parameters using a pre-trained FaceVerse decoder, so as to drive the digital human to generate emotion-enhanced facial animations.

7. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the digital human expression generation method based on multimodal feature fusion and emotion enhancement as described in any one of claims 1 to 5 when executing a computer program.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the digital human expression generation method based on multimodal feature fusion and emotion enhancement as described in any one of claims 1 to 5.

Images