Facial action unit based emotion controllable speaker face video generation method, system, device and medium

Abstract

The present application relates to a kind of facial action unit-based emotion controllable speaker face video generation method, system, equipment and medium, comprising: driving audio and the prompt text information associated with AU are input into audio large language model, semantic-emotion information in driving audio is extracted by space-time sparse strategy, and sparse AU sequence is generated based on "emotion-AU" gradual reasoning mechanism;Based on the sparse AU sequence generated, context-aware AU embedding sequence is constructed, and target face image, driving audio feature are used as the generation condition of controllable video, input video generation model, and the video generation model is guided to generate synchronous, natural and emotion expressive speaker face video using AU decoupling guide strategy.Therefore, the present application can be widely applied in computer vision, deep learning, large language model and multi-modal generation technical field.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, large language models and multimodal generation technology, specifically to a method, system, device and medium for generating emotion-controlled speaking face videos based on facial action units (AUs), which can be applied to scenarios such as virtual humans, film and television production and human-computer interaction. Background Technology

[0002] With the rapid development of technologies such as digital humans, film and television production, and human-computer interaction, audio-driven speaking face generation has become one of the important research directions in the field of computer vision. This task takes audio signals and target face images as input to generate face videos that are highly synchronized with the audio content in time, while maintaining the speaker's identity and displaying natural and realistic expressions. Currently, mainstream methods have made significant progress in lip-syncing and identity preservation, and are widely used in scenarios such as virtual hosts, virtual customer service, and automated film dubbing.

[0003] Existing methods generally rely on coarse-grained emotion labels (such as "happy" and "angry") to control the generation process. However, in scenarios that pursue higher emotional expression, this approach often fails to capture subtle facial muscle dynamics, resulting in a lack of naturalness and variation in expressions. The expressions are stiff, repetitive, and fail to meet the requirements of realism and emotional resonance.

[0004] Facial Action Units (AUs) are a crucial component of facial expression coding systems, used to characterize the contraction and movement of local muscles, thus describing facial expressions at a finer granular level. According to the Facial Action Coding System (FACS) proposed by Ekman et al., facial expressions can be decomposed into 44 independent AUs, each corresponding to a specific muscle activity. These facial action units can appear individually or combine to form complex expressions. To simplify the modeling process, subsequent studies have redefined AUs and selected 24 more representative action units for expression modeling. Compared to emotion labels, AUs possess higher expressive dimensionality and composability, and can more accurately reflect the subtle emotional changes contained in speech.

[0005] While facial action units (FAs) have a natural advantage as emotion control signals, they still face key challenges in audio-driven speech face generation. First, publicly available audio-AU pairing data is extremely limited. Furthermore, AU activation is often sparse and small in amplitude, making accurate and efficient regression of AU sequences from speech signals difficult. Currently, there is no mature method to directly infer AU signals from audio, a problem that severely restricts the application of AUs in this field. Second, how to effectively incorporate AUs as control conditions into video generation models, maintaining controllable facial expressions while also considering lip-sync, identity consistency, and overall visual quality, still lacks a reasonable structural design.

[0006] Therefore, there is an urgent need to propose a new method to improve the generation quality while ensuring the accuracy and controllability of the audio and image modalities, and to build a fine-grained semantic bridge between the audio and image modalities, so as to achieve high-quality speaking face video generation with realism and emotional details. Summary of the Invention

[0007] To address the aforementioned issues, the present invention aims to provide a method, system, device, and medium for generating emotion-controlled speaking facial videos based on facial action units. It is the first to utilize a large audio language model to directly regress AU sequences from the original speech, and uses AUs as intermediate representations to drive a diffusion model to generate facial videos, thereby achieving speaking faces with delicate and controllable emotions, accurate lip-sync, and consistent identity.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for generating emotion-controlled speaking face videos based on facial motion units, comprising: The driving audio and AU-related prompt text information are input into the audio large language model. Semantic-emotional information in the driving audio is extracted through a spatiotemporal sparsity strategy, and a sparse AU sequence is generated based on the "emotion-AU" stepwise reasoning mechanism. Context-aware AU embedding sequences are constructed based on the generated sparse AU sequences. These, along with the target face image and driving audio features, serve as the conditions for generating controllable videos. The video generation model is then input into the model, and an AU decoupling guidance strategy is used to guide the model to generate synchronized, natural, and emotionally expressive speaking face videos.

[0010] Secondly, the present invention provides an emotion-controlled speaking face video generation system based on facial motion units, comprising: The AU sequence generation module is used to input the input audio and AU-related prompt text information into the audio large language model, extract semantic-emotion information from the input audio through a spatiotemporal sparsity strategy, and generate sparse AU sequences based on the "emotion-AU" stepwise reasoning mechanism. The video generation module is used to construct a context-aware AU embedding sequence based on the generated sparse AU sequence. This sequence, along with the target face image and input audio features, serves as the conditions for generating controllable videos. The video generation model is then input into the AU decoupling guidance strategy, which guides the diffusion model to generate synchronized, natural, and emotionally expressive speaking face videos.

[0011] Thirdly, the present invention provides a computer-readable storage medium for storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any method.

[0012] Fourthly, the present invention provides a computing device comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing any method.

[0013] The present invention has the following advantages due to the adoption of the above technical solutions: (1) This invention pioneered the fine-grained regression of audio to facial action units (AUs). For the first time, this invention introduces a large audio language model to perform semantic understanding of audio signals and achieves the automatic regression of fine-grained AU expressions from natural speech through a stepwise reasoning strategy of "emotion-action unit". This method breaks through the existing expression method that relies on coarse-grained emotion labels, establishes a continuous mapping relationship between audio and facial expressions, and improves the naturalness and richness of facial emotion expression.

[0014] (2) An efficient spatiotemporal sparsity coding mechanism is proposed to improve the representational modeling capability of AU. This invention designs a spatiotemporal joint sparsity strategy, which extracts "index-intensity" pairs through threshold sparsity in space and compresses the sequence length by downsampling in time, so that the original dense AU signal can be transformed into a structured, low-dimensional and model-friendly discrete semantic sequence. This strategy not only effectively reduces data redundancy, but also is more suitable for the language-based modeling process of large models, improving the accuracy and stability of AU generation.

[0015] (3) Constructing a structure-prior-driven AU-controlled face video generation framework. In the video generation stage, this invention utilizes low-frame-rate AU sequences for linear interpolation and constructs facial structure priors through keypoint or mesh rendering, guiding the diffusion model to generate spatially consistent and topologically reasonable simulated face animations. By introducing a context-aware AU embedding mechanism and an "AU-vision" cross-attention fusion module, the ability of AU control to express facial dynamics is further enhanced, achieving video output with greater emotional realism and individual consistency.

[0016] (4) An adjustable AU decoupling guidance strategy is proposed. This invention introduces an AU decoupling guidance mechanism in the diffusion model inference stage, by setting the AU control strength ( ) and reference image control intensity ( The weight parameters are used to achieve a flexible balance between emotional expression and visual quality. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a flowchart of the emotion-controlled speaking face video generation method based on facial motion units provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the generated results in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0020] Existing audio-driven face video generation methods mostly rely on coarse-grained emotion tags, lacking detailed modeling of micro-expressions and emotional continuity, making it difficult to generate videos with high emotional realism and facial dynamic consistency. Therefore, some embodiments of this invention provide an emotion-controlled speaking face video generation method based on facial motion units, aiming to achieve automatic synthesis from speech audio into realistic face videos with nuanced emotional expression.

[0021] This invention employs a two-stage generative framework, primarily consisting of audio-driven AU representation modeling and AU-controlled video generation. In the first stage, an audio large language model is used to extract semantic-emotional information from the audio through a spatiotemporal sparsity strategy. A sparse AU sequence is generated based on an emotion-AU stepwise inference mechanism, addressing the issues of limited AU representation and scarce supervision. In the second stage, the generated sparse AU sequence is interpolated and aligned to the target frame rate and mapped to a structured facial representation, guiding a diffusion model to generate synchronized, natural, and emotionally expressive speaking face videos. An AU decoupling guidance strategy is introduced during inference, balancing visual quality and expressive accuracy by adjusting the guidance strength weights between AUs and other conditions. Experiments show that this invention outperforms existing technologies in terms of emotional realism, lip-sync, and character consistency, demonstrating strong versatility and practical value.

[0022] Correspondingly, in other embodiments of the present invention, an emotion-controlled speaking face video generation system, device, and medium based on facial motion units are provided.

[0023] Example 1 like Figure 1 As shown, the present invention provides a method for generating emotion-controlled speaking face videos based on facial motion units, comprising the following steps: (1) First stage: Input the driving audio and the prompt text information related to AU into the audio big language model, extract the semantic-emotion information in the driving audio through the spatiotemporal sparsity strategy, and generate sparse AU sequence based on the "emotion-AU" stepwise reasoning mechanism; (2) Second stage: Construct context-aware AU embedding sequence based on the generated sparse AU sequence, and use it together with the target face image and driving audio features as the generation conditions of controllable video, input it into the video generation model, and use the AU decoupling guidance strategy to guide the video generation model to generate synchronous, natural and emotionally expressive speaking face video.

[0024] Furthermore, step (1) above includes the following steps: (1.1) Preprocess the original video to obtain the driving audio and target video frames; (1.2) Input the target video frame into the pre-built AU regression model to obtain the AU sequence pseudo-label vector; (1.3) The AU sequence pseudo-label vector is sparsified by the spatiotemporal sparsification strategy to obtain the sparse AU sequence pseudo-label vector; (1.4) The sparse AU sequence pseudo-label vector, driving audio, and AU-related prompt text information are fed into the Audio-Language Model (ALM). The Audio-Language Model uses the "emotion-AU" step-by-step inference mechanism to generate sparse AU sequences. .

[0025] Furthermore, in step (1.1) above, specifically, the preprocessing of the original video includes: reading the target face image and driving audio from the original video and extracting the target video frame; resampling the driving audio to 16kHz and extracting the Mel spectrum as audio features; constructing supervised pairs (audio segment, AU time sequence, target video frame series) based on identity-independent partitioning to provide consistent temporal alignment and control conditions for subsequent steps.

[0026] Furthermore, in step (1.2) above, the AU regression model can be trained using any mainstream backbone network. In this embodiment, the backbone network used is ResNet50, whose input is video frame images and output is a 24-dimensional continuous intensity AU sequence.

[0027] Furthermore, in step (1.3) above, in order to reduce the encoding complexity of the facial action unit sequence and improve its structural alignment with the audio large language model, this invention proposes a spatiotemporal sparsity strategy, which mainly includes two aspects: spatial sparsity and temporal compression.

[0028] (1.3.1) Spatial sparsity: The original AU sequence pseudo-labels of each frame image are filtered based on element intensity according to a preset threshold to form a sparse representation. .

[0029] In this embodiment, the original dense AU sequence pseudo-labels for each frame of image are... The 24 dimensions correspond to the 24 AUs defined in the FEAFA+ public dataset. Filtering is performed based on the element strength, retaining only those above a threshold. The dimensional information is transformed into a set of "index-strength" pairs to form a sparse representation. : This method can effectively remove muscle action dimensions with no significant activation, retain key information, thereby compressing the representation length and improving modeling efficiency.

[0030] (1.3.2) Time compression: based on time sampling factor , for sparse representation Equal-interval sampling is performed to obtain sparse AU sequences aligned with the audio large language model.

[0031] Considering the difference between the AU sequence and the speech frame rate, and the fact that the original frame-level AU sequence is quite long and difficult to model directly, this invention introduces a time sampling factor. The AU sequence is sampled at equal intervals, that is, every... Each frame retains an AU representation. This strategy significantly reduces the length of the AU sequence, making it easier to align with large audio language models.

[0032] Through the combined strategy of spatial sparsity and temporal compression, the overall length of the AU sequence is significantly reduced, making it easier to use as the input sequence for large audio language models. This effectively alleviates the burden on the model to process long sequences, while preserving key facial dynamic information, thus achieving a more compact and efficient action unit modeling process.

[0033] Furthermore, in step (1.4) above, to improve the accuracy and stability of audio-to-facial action unit sequence regression, this invention proposes an AU regression method based on an "emotion-AU" chain inference mechanism. This method draws on the coarse-to-fine generation paradigm of Chain-of-Thought (CoT), first inferring the emotion category in the high-level semantic space of speech, and then generating the corresponding AU sequence under the guidance of emotion semantics, achieving hierarchical modeling from speech to muscle movement. Specifically, it includes two aspects: emotion prediction and facial action unit regression. (1.4.1) Sentiment Prediction: The driving audio and AU-related prompt text information are input into the audio big language model. Under internal guidance, the audio big language model generates coarse-grained sentiment labels corresponding to the driving audio. Its category set is: .

[0034] In this embodiment, the emotion tag generation process of the audio big language model is used as a priori guidance to provide global semantic information for subsequent AU sequence generation. The audio big language model completes the multi-class classification task through autoregression, and the emotion classification module is trained using the cross-entropy loss function. ,in, The number of emotion categories is 8. For real labels, To predict probabilities.

[0035] (1.4.2) Facial motion unit regression: converting coarse-grained emotion labels As an emotional context, the audio big language model generates sparse AU sequences. .

[0036] In emotion tags After being predicted, the audio big language model further generates sparse AU sequences within the emotional context. Each time step The AU representation is expressed in discrete form as: This discrete format can be used as a natural language symbolization input, naturally embedded into the generation stream of the discrete language model, thus aligning with the model's pre-training task and improving learning efficiency.

[0037] Furthermore, step (2) above includes the following steps: (2.1) Align the sparse AU sequence output in step (1) with the target video frame rate and map it into a structured two-dimensional face representation; (2.2) Generate context-aware AU embedding sequences based on structured two-dimensional facial representation; (2.3) The context-aware AU embedding sequence, the target face image and the input audio features are used as input to the video generation model, and an AU-visual fusion mechanism based on cross-attention is introduced into the video generation model to realize the interaction between AU information and visual features. (2.4) Use the AU decoupling guidance strategy to guide the video generation model to generate synchronous, natural and emotionally expressive speaking face videos.

[0038] Furthermore, in step (2.1) above, to achieve precise and controllable face generation driven by facial motion units, it is necessary to align the low frame rate AU sequence output in step (1) with the target video frame rate and construct a structured spatial representation. This step includes two sub-tasks: temporal alignment and spatial structured modeling, specifically: (2.1.1) Time alignment: The sparse AU sequence is interpolated and upsampled over time to obtain an AU sequence aligned with the target video frame rate. .

[0039] Since the sparse AU sequence generated in step (1) typically has a low temporal resolution (e.g., 5 frames / second), while the target video frame rate is typically 25 frames / second, the original sparse AU sequence needs to be interpolated and upsampled over time to achieve frame-level synchronization control. In this embodiment, a linear interpolation method is used. Let the original sparse AU sequence be:

[0040] Then, for each dimension of the AU sequence representation in the sparse AU sequence, a time extension is performed using a linear interpolation function, resulting in:

[0041] in, , The target face video frame rate, and , The sampling factor; ; This represents the linear weights between the two sampling points. After interpolation, a complete frame-level AU sequence of the target face video is obtained. .

[0042] (2.1.2) Spatial structured modeling: The AU sequence generated in step (2.1.1) is aligned with the target video frame rate. Mapped to a structured two-dimensional facial representation.

[0043] Although the AU representation itself is a one-dimensional muscle activation intensity vector, it essentially corresponds to the spatial motion distribution of facial muscle groups. To better guide video generation models in synthesizing facial images with anatomical consistency and expression accuracy, this invention further maps the AU representation to a structured two-dimensional facial representation, i.e., explicitly constructing the facial topology. This invention provides two optional implementation methods: one based on facial keypoints and the other based on mesh rendering.

[0044] Furthermore, in step (2.2) above, although the AU sequence vector of a single frame... While facial expressions can reflect the current state of muscle activation, due to the continuity and dynamic characteristics of facial expressions, it is difficult to model the temporal smoothness and contextual relevance of real facial changes using only single-frame information. Therefore, this invention proposes a context-aware AU representation method, which encodes the AU sequence of consecutive frames into a temporally consistent embedding representation by constructing a local temporal window. The specific steps include: constructing a local time window, designing a sequence encoder, and outputting the embedded sequence.

[0045] (2.2.1) Construction of local time windows: Set the length of the time window to... For the current frame Sampling the frame before and after Frames, constructing local sequences :

[0046] (2.2.2) Sequence Encoding Design: Temporal ConvNet is used as the context modeler to encode the local sequences in step (2.2.1). Mapped to a fixed-length vector: ,in, This is the dimension embedded in the AU, typically 256 or 512.

[0047] (2.2.3) Embedded sequence output: Repeat steps (2.2.1) to (2.2.2) to iterate through each time frame. This yields the complete context-aware AU embedding sequence: .

[0048] Furthermore, in step (2.3) above, in order to embed the context-aware AU into the sequence By effectively incorporating the generation process of the video generation model, this invention designs an AU-visual fusion strategy based on a cross-attention mechanism, which realizes the interaction between AU information and visual features in each noise reduction step of the diffusion model.

[0049] In each layer of the UNet backbone network, a cross-attention mechanism module is inserted, specifically as follows: ， in For the first Feature map of layer , The AU embedding sequence is used. In the cross-attention module of the diffusion model, a "query-key-value" mechanism is adopted to realize the information interaction between the AU conditions and visual features. The visual features serve as the "query," and the AU representations serve as the "key-value" input, achieving cross-modal information fusion. To avoid disrupting the pre-trained model structure, the cross-attention module is zero-initialized and trained separately, while the remaining backbone parameters remain frozen.

[0050] Specifically, the UNet backbone network includes a downsampling encoder, intermediate blocks, and an upsampling decoder, with the downsampling encoder and decoder connected via skip connections. The overall input to the UNet backbone network includes the noise latent variable at the current time step. Time step And conditional coding (including AU embedding sequences, etc.); its overall output is the predicted noise residual. The AU cross-attention module is embedded in each layer of UNet, receiving intermediate feature maps from the current layer. With AU embedding sequence The calculated fusion feature map is used as input. It will be directly returned to the backbone network, serving as input for the next layer or participating in subsequent convolution operations, and finally output by UNet as a prediction result for denoising.

[0051] Furthermore, in step (2.4) above, to balance AU control accuracy and overall image quality during the actual generation process, this invention proposes an AU decoupling guidance mechanism. This mechanism achieves a dynamic balance between control information and image naturalness by adjusting the guidance weights of different conditions. The core idea of ​​this strategy is to introduce multiple guidance terms into the standard conditional diffusion generation process, including AU conditions, reference image conditions, and empty conditions. The differences between these conditions guide the generator to converge towards the target distribution, thereby achieving controllable enhancement of specific features. Let the noise prediction function of the diffusion model be... ,in This represents the current latent variable input. These are the model parameters. This decoupling guidance strategy combines multiple conditional predictions in the following way:

[0052]

[0053] in, This represents the potential representation of the current diffusion time step; This represents the noise prediction function of the diffusion model under different conditions; An empty condition indicates unconditional generation without introducing any control information; Provides facial expression information for the control signals embedded in the AU; The face structure control signal is derived from the reference image; To guide the weights and control the intensity of the influence of AU conditions on the generated results; The weights are guided by the reference image to control the degree to which the consistency of the person's identity is maintained.

[0054] This strategy essentially uses interpolation to balance the relative effects of AU control and other conditions. Users can set different bootstrap weights according to their specific application needs. and This allows for flexible control in two dimensions: the intensity of emotional control and its improvement. It can generate more expressive results; identity fidelity control is improved. This allows for better preservation of the facial structure and style of the figures in the reference image. When AU conditions are dominant (such as in scenes where the virtual human's emotions are driven), the accuracy should be appropriately increased. When character consistency is critical (e.g., in scenarios involving voice cloning and facial recognition), enhancement should be prioritized. .

[0055] Example 2 This invention proposes an emotion-controlled speaking face video generation method based on facial action units. This method, for the first time, directly regresses the original speech into an AU sequence and uses the AU sequence as an interpretable intermediate control condition to drive the video generation model to synthesize speaking face videos with nuanced emotions, synchronized lip movements, and consistent identity.

[0056] To facilitate understanding and implementation, this embodiment provides a detailed description of the technical solution of the present invention. In this embodiment, the hardware configuration for executing the algorithm is as follows: an Intel i9 CPU and four NVIDIA A100 graphics cards with 40GB of RAM each are used for the first and second stages of training. The first stage training requires approximately 96 GPU hours, and the second stage training for each pedestal model requires approximately 48 GPU hours. The software configuration is as follows: the computer operating system is Ubuntu 20.04, CUDA version 12.4, and the neural network framework used is PyTorch, version 2.1.

[0057] The data used in this invention comes from two publicly available emotion-based speech-face video datasets: MEAD and CREMA, both of which provide standard identity segmentation training / testing protocols and are widely used in academic face synthesis and multimodal modeling tasks. To ensure consistency in multimodal input formats, the original video and audio data are first uniformly preprocessed, including: normalizing the video to 25 frames per second (fps) and unifying the image resolution to 512×512 pixels; resampling the audio to 16 kHz and using parameters with a window size of 640 samples and a frame shift of 640 samples to extract Mel-spectrograms as audio feature representations. During data construction, the corresponding audio segments, reference face images, facial action unit (AU) sequences, and target video frame sequences are extracted from each video segment to form complete training samples, which support the proposed two-stage modeling framework of "audio→AU→video". In this process, the AU sequence serves as the supervisory signal for the first-stage input audio to the AU regression task, while the reference face image, audio features, and AU embedding representation together constitute the conditional input for the second-stage controllable video generation, thereby achieving path alignment and semantic collaboration among multiple modalities at the training level.

[0058] Since existing MEAD and CREMA datasets do not directly provide true labels for facial action units (AUs), this invention trains a high-precision AU regression model based on the FEAFA+ public dataset to obtain reliable AU supervision signals. The FEAFA+ public dataset contains a large number of real images and their corresponding manually labeled AU intensity information, providing a reliable training foundation for AU estimation. Specifically, a ResNet50 is used as the backbone network, and a regression head is designed to predict the 24-dimensional AU intensity vector. On the validation set, the mean squared error is only 0.0034, demonstrating high fitting accuracy and generalization ability. Subsequently, this model is applied to the MEAD and CREMA datasets to perform AU label regression on each frame of image, and combined with manual sampling verification to ensure its high reliability in terms of emotional expression and muscle movement. The aforementioned regression labels ultimately serve as training supervision signals and conditional inputs to support the implementation process of audio-driven AU estimation and video generation tasks in this invention.

[0059] This embodiment is based on the two-stage emotion-controlled speaking face video generation framework AUHead, which combines spatiotemporal AU sparsity, coarse-to-fine reasoning, facial action unit to 2D facial representation mapping, context-aware AU embedding, "AU-vision" cross-modal interaction adapter, and decoupled guided reasoning mechanism. The overall structure uses a pre-trained video generation model as the backbone, with reference face images and audio information as input. In the first stage, facial action unit (AU) sequences are regressed from audio. In the second stage, the AU sequences are used as the core control conditions to generate speaking face videos with nuanced emotions, synchronized lip movements, and consistent identity, achieving end-to-end controllable synthesis from audio to video.

[0060] The specific implementation steps include: 1. Input regularization and pairing construction Read the target face image and driving audio from the original video file; extract the target video frame from the original video; resample the driving audio to 16 kHz and extract the Mel spectrum as audio features; construct supervised pairs (audio segment, AU time sequence, target video frame series) based on identity independence to provide consistent temporal alignment and control conditions for subsequent steps.

[0061] 2. Regressing AU sequences from audio This stage uses audio from the MEAD dataset as input and frame-level AU vectors obtained by inferring on the MEAD dataset using an AU regression model trained on the FEAFA+ dataset as supervised "pseudo-labels". To adapt to the context length constraints of large audio language models, a spatiotemporal sparse encoding mechanism and formatted text prompts are proposed, and lightweight fine-tuned on Qwen-Audio-Chat V1 using LoRA. Training adopts a coarse-to-fine inference approach of "emotion first, AU later", where the emotion category is given first in a sentence output, followed by the sparse AU sequence.

[0062] 2.1 Pseudo-annotation generation and alignment 2.1.1 Training the AU regression model: An AU regression model is trained on the FEAFA+ dataset, outputting a 24-dimensional continuous intensity (range [0,1]) AU sequence. The network structure of the AU regression model can be any mainstream backbone network (ResNet50 is used in this embodiment), with its final linear layer replaced by a 24-dimensional output layer, and a HardSigmoid activation function applied. The model is optimized using Adam, employing a cosine learning rate scheduling (initial learning rate is 1e). -4 (Batch size = 256), 100 rounds of data were trained using the mean squared error loss function.

[0063] 2.1.2 Pseudo-label generation: The AU regression model is applied to the video inference in the MEAD dataset frame by frame to obtain aligned AU sequence pseudo-labels.

[0064] 2.2 Spatiotemporal Sparsity Coding Mechanism To adapt to the context length of large audio language models and highlight key muscle activation signals, the frame-level AU dense vectors are first spatially sparsified: only "index-intensity" pairs above a threshold are retained, and dimensions with insignificant activation are removed, resulting in a compact sparse frame representation. ,in , .

[0065] For example: a dense action unit (AU) vector is After sparsification, it is encoded into a sparse set. This means that only the dimension indices with non-zero values ​​and their corresponding intensity values ​​are retained. This sparse coding method can significantly shorten the representation length of AU sequences, reducing the sequence length by an average of about 80.95% in experiments.

[0066] Then, time compression is performed: the AU sequence is compressed by a fixed step size. Equal-interval extraction significantly shortens the time sequence length, making it easier for the model to process as a discrete language sequence.

[0067] 2.3 Audio Frame Segmentation and Cue Construction A conversational prompt is constructed, explicitly defining the fixed order and meaning of 24 AUs in the text prompt. A sparse "index-intensity" list is agreed upon as the output format for each frame (intensity uses two decimal places for stable decoding and verification). The prompt organization follows the structure of "analyzing emotion first, then outputting the AU sequence," ensuring that the audio large language model has a consistent high-level emotional context before providing the AUs.

[0068] 2.4 Model Configuration and Fine-tuning This invention selects Qwen-Audio-Chat V1 as the audio large language model, enabling its audio front-end and multimodal alignment layer. LoRA fine-tuning positions include: Q / K / V projection and output projection of multi-layer attention, and insertion of LoRA adapters for upper / lower projections of intermediate multi-layer perceptrons. AdamW (lr=1×10) is used. 4 Linear warm-up to 10% of training steps followed by cosine annealing.

[0069] Predict sentiment labels first The loss function is ,in, The number of emotion categories is 8. For real labels, To predict probabilities, in the context of "sentiment-AU", we model the next prediction sequence. The loss function is... ,in As the emotional prior from the previous step, A represents the driving audio. The training parameters consist only of the LoRA adapter layer and the output header; the backbone remains frozen, and AU sequences are generated autoregressively during inference, driven by audio conditions. This invention uses four NVIDIA A100 GPUs to complete this stage of training, approximately 24 GPUs; the learning rate is 1×10⁻⁶. -4 Other training strategies can be configured according to conventional deep learning practices.

[0070] 3. Controllable face video generation based on AU sequences The low frame rate AU sequence output from step 2 is denoted as... , The target frame rate of the video is ,in The sampling factor is used; for each dimension of AU, a linear interpolation function is used for time extension:

[0071] in, , ; This represents the linear weights between the two sampling points. After interpolation, a complete video frame-level AU sequence is obtained. This ensures time alignment with subsequent video synthesis models. After interpolation, a complete video frame-level AU sequence is obtained. Furthermore, the one-dimensional AU vector is mapped to a structured two-dimensional facial representation, providing an explicit facial topological prior. This invention offers two optional forms: keypoint rendering and differentiable mesh rendering. This step significantly enhances the spatial interpretability and generative controllability of the AU.

[0072] 3.1 Context-Aware AU Embedding To characterize the continuity and fine-grained dynamics of facial expressions, an AU embedding constructed using local temporal windows is introduced. For the first... Frame, in length Concatenate within a window to construct a local sequence:

[0073] Embedded by one-dimensional temporal convolution ,in The dimension embedded for AU. Iterate through each time step. This allows us to obtain the complete context-aware AU embedding sequence: In this embodiment, the window size is set to 5 (i.e., n=2), which balances short-term dynamics and temporal smoothing.

[0074] 3.2 AU-Visual Fusion Mechanism Based on Cross-Attention In each layer of the pre-trained UNet backbone network, an AU adapter, namely a Cross-Attention module, is inserted, specifically as follows: ,in For the first Feature map of layer , For AU embedding sequences. Let visual latent variables... right Across attention, an adaptive AU control signal is injected at each denoising step.

[0075] 3.3 AU Decoupling Guidance Strategy In the inference phase, to balance AU control accuracy and overall image quality, this invention introduces multiple guiding terms during the standard conditional diffusion generation process, including: AU condition, reference image condition, and null condition. The differences between these conditions guide the generator to converge toward the target distribution, thereby achieving controllable enhancement of specific features. Let the noise prediction function of the diffusion model be... ,in This represents the current latent variable input. These are the model parameters. This decoupling guidance strategy combines multiple conditional predictions in the following way:

[0076]

[0077] in, This represents the potential representation of the current diffusion time step; This represents the noise prediction function of the diffusion model under different conditions; An empty condition indicates unconditional generation without introducing any control information; Provides facial expression information for the control signals embedded in the AU; The face structure control signal is derived from the reference image; To guide the weights and control the intensity of the influence of AU conditions on the generated results; The reference image guides the weighting, controlling the degree to which the consistency of the person's identity is maintained. In this invention, the weighting is... and Set them all to 3.5.

[0078] 3.4 Model Configuration and Fine-tuning Phase two used Hallo V1 and MEMO as the basis; the experiment was reproduced on both with the same data partitioning and settings. The learning rate was 5×10⁻⁶. 6(Hallo V1) and 1×10 5 (MEMO); Both models were trained on a 4×A100 for 12 GPU-hours, with a context window of 5. The loss function remains the same as that used in Hallo V1 and MEMO. The remaining hardware and software environment can be set up according to conventional deep learning configurations and does not constitute a limitation on the implementation of this invention.

[0079] To verify the effectiveness and practicality of the proposed method, results on the MEAD and CREMA datasets are presented below. Table 1 shows the detection results of the instances on the test set. Objective metrics include PSNR, SSIM, FID (image quality and perceived realism), M-LMD (lip-sync), F-LMD (structure preservation), and ACC. emo (Emotional consistency). On MEAD and CREMA, this invention outperforms existing methods in most metrics, demonstrating higher visual fidelity, lip geometry accuracy, and facial expression subtlety.

[0080] Table 1 shows the detection results of the instances on the MEAD and CREMA datasets.

[0081] As shown in Table 1, the present invention was comprehensively compared with state-of-the-art methods on the MEAD and CREMA datasets, and the method consistently outperformed previous methods on most metrics. In particular, higher PSNR and SSIM scores were achieved, indicating improved visual fidelity and structural consistency, and lower FID, suggesting that AU-based expression modeling enhances the perceptual realism of the generated videos. Furthermore, the present invention achieved lower M-LMD and F-LMD scores, indicating more accurate lip geometry and better preservation of facial structure. These results demonstrate that AU conditioned reflexes enhance the fidelity of mouth movements and facial expression details.

[0082] Example 3 The above-described embodiment 1 provides a method for generating emotion-controlled speaking face videos based on facial motion units. Correspondingly, this embodiment provides a system for generating emotion-controlled speaking face videos based on facial motion units. The system provided in this embodiment can implement the emotion-controlled speaking face video generation method based on facial motion units of embodiment 1. The system can be implemented through software, hardware, or a combination of both. For example, the system may include integrated or separate functional modules or units to perform the corresponding steps in the methods of embodiment 1. Since the system in this embodiment is basically similar to the method embodiment, the description process in this embodiment is relatively simple. Relevant details can be found in the description of embodiment 1. The system embodiment provided in this embodiment is merely illustrative.

[0083] The emotion-controlled speaking face video generation system based on facial motion units provided in this embodiment includes: The AU sequence generation module is used to input the input audio and AU-related prompt text information into the audio large language model, extract semantic-emotion information from the input audio through a spatiotemporal sparsity strategy, and generate sparse AU sequences based on the "emotion-AU" stepwise reasoning mechanism. The video generation module is used to construct a context-aware AU embedding sequence based on the generated sparse AU sequence. This sequence, along with the target face image and input audio features, serves as the conditions for generating controllable videos. The video generation model is then input into the AU decoupling guidance strategy, which guides the diffusion model to generate synchronized, natural, and emotionally expressive speaking face videos.

[0084] Example 4 This embodiment provides a processing device corresponding to the emotion-controlled speaking face video generation method based on facial motion units provided in Embodiment 1. The processing device can be a client-side processing device, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Embodiment 1.

[0085] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to communicate with each other. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the emotion-controlled speaking face video generation method based on facial motion units provided in Embodiment 1.

[0086] Preferably, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0087] Preferably, the processor can be any type of general-purpose processor such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation herein.

[0088] Example 5 The emotion-controlled speaking face video generation method based on facial motion units in Embodiment 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the emotion-controlled speaking face video generation method based on facial motion units described in Embodiment 1 are loaded.

[0089] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating emotion-controlled speaking face videos based on facial motion units, characterized in that, include: The driving audio and AU-related prompt text information are input into the audio large language model. Semantic-emotional information in the driving audio is extracted through a spatiotemporal sparsity strategy, and a sparse AU sequence is generated based on the "emotion-AU" stepwise reasoning mechanism. Context-aware AU embedding sequences are constructed based on the generated sparse AU sequences. These, along with the target face image and driving audio features, serve as the conditions for generating controllable videos. The video generation model is then input into the model, and an AU decoupling guidance strategy is used to guide the model to generate synchronized, natural, and emotionally expressive speaking face videos.

2. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 1, characterized in that, The process involves inputting the driving audio and AU-related prompt text information into an audio large language model, extracting semantic-emotional information from the driving audio using a spatiotemporal sparsity strategy, and generating a sparse AU sequence based on an "emotion-AU" stepwise inference mechanism, including: The original video is preprocessed to obtain the driving audio and the target video frames; Input the target video frame into the pre-built AU regression model to obtain AU sequence pseudo-labels; A spatiotemporal sparsity strategy is used to sparsify the pseudo-labels of AU sequences, resulting in sparse AU sequence pseudo-labels. The sparse AU sequence pseudo-labels, driving audio, and AU-related prompt text information are fed into the audio big language model, which then uses a "emotion-AU" step-by-step reasoning mechanism to generate sparse AU sequences.

3. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 2, characterized in that, The method of sparsifying AU sequence pseudo-labels using a spatiotemporal sparsity strategy to obtain sparse AU sequence pseudo-labels includes: The pseudo-labels of the AU sequence of each frame image are filtered based on element intensity according to a preset threshold to form a sparse representation. Based on time sampling factor By sampling the sparse representation at equal intervals, sparse AU sequence pseudo-labels aligned with the audio large language model are obtained.

4. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 2, characterized in that, include: The process involves feeding sparse AU sequence pseudo-labels, driving audio, and AU-related prompt text information into an audio large language model. The audio large language model then uses a "sentiment-AU" stepwise inference mechanism to generate sparse AU sequences, including: The driving audio and AU-related prompt text information are input into the audio language model, which generates coarse-grained emotion tags corresponding to the driving audio under internal guidance. Using coarse-grained emotion tags as the emotional context, the audio big language model generates sparse AU sequences.

5. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 2, characterized in that, The process involves constructing a context-aware AU embedding sequence based on the generated sparse AU sequence, which, together with the target face image and input audio features, serves as a condition for generating controllable video. This is input into the video generation model, and an AU decoupling guidance strategy is used to guide the diffusion model to generate synchronized, natural, and emotionally expressive speaking face videos. Align the sparse AU sequence with the target video frame rate and map it into a structured two-dimensional facial representation; Based on structured 2D facial representation, a context-aware AU embedding sequence is generated; The context-aware AU embedding sequence, the target face image, and the input audio features are used as inputs to the video generation model. An AU-visual fusion mechanism based on cross-attention is introduced into the video generation model to realize the interaction between AU information and visual features. The AU decoupling guidance strategy is used to guide the video generation model to generate synchronous, natural, and emotionally expressive speaking face videos.

6. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 5, characterized in that, The generation of context-aware AU embedding sequences based on structured two-dimensional facial representation includes: Set the time window length to For the current frame Sampling the frame before and after Frames, constructing local sequences ; Temporal convolutional networks are used as context modelers to model local sequences. Mapped to a fixed-length vector: ,in, The dimension embedded in AU; Iterate through each time frame This yields the complete context-aware AU embedding sequence: .

7. The method for generating emotion-controlled speaking face videos based on facial motion units as described in claim 5, characterized in that, The AU decoupling guidance strategy combines multiple conditional predictions in the following manner: in, This represents the potential representation of the current diffusion time step; This represents the noise prediction function of the diffusion model under different conditions; An empty condition indicates unconditional generation without introducing any control information; Provides facial expression information for the control signals embedded in the AU; The face structure control signal is derived from the reference image; To guide the weights and control the intensity of the influence of AU conditions on the generated results; The weights are guided by the reference image to control the degree to which the consistency of the person's identity is maintained.

8. A system for generating emotion-controlled speaking facial videos based on facial motion units, characterized in that, include: The AU sequence generation module is used to input the input audio and AU-related prompt text information into the audio large language model, extract semantic-emotion information from the input audio through a spatiotemporal sparsity strategy, and generate sparse AU sequences based on the "emotion-AU" stepwise reasoning mechanism. The video generation module is used to construct a context-aware AU embedding sequence based on the generated sparse AU sequence. This sequence, along with the target face image and input audio features, serves as the conditions for generating controllable videos. The video generation model is then input into the AU decoupling guidance strategy, which guides the diffusion model to generate synchronized, natural, and emotionally expressive speaking face videos.

9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 7.

10. A computing device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 7.

Images