A multi-modal facial response generation method

Abstract

The application provides a multi-modal facial response generation method, and belongs to the technical field of facial response generation. The method comprises the following steps: acquiring speaker audio, text and video multi-modal features and encoding, and aligning the time dimensions of the three types of features; enhancing the encoded features through time sequence self-attention, six-direction cross-attention, adaptive fusion and post-processing to obtain deep fusion features; constructing a shared and private double-layer decoupled latent space, and obtaining a joint latent representation through decoupled constraint, verification and staged sampling; outputting refined 3DMM facial parameters and listener emotional attributes through a four-layer generation architecture with FiLM emotion modulation. The application solves the pain points of insufficient multi-modal information utilization and insufficient long sequence modeling of traditional technologies, can generate context-adaptive and emotion-consistent facial responses, and is suitable for human-computer interaction, virtual reality and other fields.

Description

Technical Field

[0001] This invention relates to the field of facial response generation technology, and in particular to a multimodal facial response generation method. Background Technology

[0002] In cutting-edge fields such as human-computer interaction, virtual reality, affective computing, and multimedia content generation, automatic facial response generation technology has enormous application potential. As a core means of human emotional expression and the transmission of social intentions, the automatic generation of realistic, natural, and context-appropriate facial responses has become a core research topic in the field of artificial intelligence, but current technologies still face multi-dimensional technical bottlenecks.

[0003] At the level of information input and temporal modeling, existing methods mostly rely on single-modal input of audio or video, failing to fully integrate multimodal complementary information. The semantic ambiguity of audio and the limitations of video understanding are difficult to compensate for. In terms of temporal modeling, traditional Transformer models have high computational complexity and are usually difficult to support long sequence modeling of more than 256 frames, which cannot adapt to the application requirements of long-term dialogue scenarios.

[0004] At the generative modeling and cross-modal interaction levels, existing methods mostly use a single latent space to encode all generative factors, making it difficult to balance the semantic correctness of the generated results with the diversity of individual expressions; emotion guidance mostly uses simple feature splicing, lacking the ability to finely control feature channels; cross-modal interaction is only achieved through unidirectional attention or simple splicing, which cannot fully capture complementary information between modalities, ultimately resulting in significant defects in the generated facial responses in terms of contextual adaptability, emotional consistency, and naturalness. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal facial response generation method that solves the pain points of insufficient utilization of multimodal information and insufficient long sequence modeling in traditional technologies, and can generate context-adaptive and emotionally consistent facial responses.

[0006] To achieve the above objectives, the present invention provides a multimodal facial response generation method, comprising the following steps: S1. Obtain speaker multimodal feature information and encode it separately. The multimodal feature information includes audio features, text features, and video features. Align the audio features, text features, and video features to the same time step length T in the time dimension. S2. The encoded audio, text, and video features are sequentially subjected to temporal self-attention enhancement, six-way cross-modal attention enhancement, adaptive weight fusion, and post-processing of fused features to obtain deeply fused speaker fusion features. S3. Based on the fusion features, a two-layer latent space model with shared latent space and private latent space is constructed. The information of the two spaces is separated through multiple decoupling constraints. An appropriateness verification mechanism is introduced to verify and adaptively adjust the rationality of the combination of the two spaces. A phased sampling strategy is adopted to obtain the joint latent representation. S4. Based on joint latent representation and speaker video emotional features, a four-layer hierarchical generation architecture is constructed. In each layer, a FiLM feature-level linear modulation mechanism is introduced to guide emotional conditions, thereby generating coarse-grained 3DMM parameters, AU intermediate representation, refined 3DMM parameters and listener emotional attributes in sequence. S5 outputs refined 3DMM facial parameters and listener emotional attributes to complete multimodal facial response generation. The refined 3DMM parameters are used to drive the 3D facial model to generate visual animations, and the listener emotional attributes are used for high-level emotional semantic analysis of facial responses.

[0007] Preferably, in S1, the Mamba-SSM coding block is used to complete the temporal modeling of audio, text and video features. The Mamba-SSM coding block includes a selective state space layer, nonlinear activation, layer normalization and residual connection. The residual connection uses an element-wise addition method to fuse the input features and the encoded output.

[0008] Preferably, the six-way cross-modal attention enhancement in S2 is to construct a bidirectional cross-attention mechanism for text-audio, text-video, audio-text, audio-video, video-text, and video-audio. It realizes cross-modal information interaction through a learnable query, key, and value mapping matrix, and enhances the feature expression of another modality by utilizing semantic / prosodic / visual information of one modality, thus fully capturing the complementary information between the three modalities.

[0009] Preferably, in the two-layer latent space model of S3, the shared latent space encodes semantic information shared between modalities to ensure the correctness and appropriateness of the generated response, while the private latent space encodes individual-specific style information to provide diversity and personalization of the generated response; The shared latent space is obtained by reparameterizing the fused features after attention pooling and MLP estimation of mean and variance. The private latent space is obtained by reparameterizing the original features of audio, text, and video after encoding and concatenation by a dedicated encoder, pooling, and MLP estimation of mean and variance. The proprietary modal encoder includes an input mapping layer, a selective state-space modeling layer with residual connections, a nonlinearity and normalization layer, and an output mapping layer.

[0010] Preferably, the multiple decoupling constraints in S3 include mutual information minimization constraints, adversarial decoupling constraints, reconstruction decoupling constraints, and cross-modal consistency constraints, with the comprehensive decoupling loss being: ; in, To minimize the constraint loss of mutual information, In order to combat losses as a whole, To reconstruct the decoding loss, For cross-modal consistency loss, All are decoupling constraint weights; The mutual information minimization constraint uses MINE to estimate and minimize the mutual information between two potential spaces. The adversarial decoupling constraint introduces a discriminator that prevents it from predicting one space from another. The reconstruction decoupling constraint enables the two spaces to independently recover their corresponding information. The cross-modal consistency constraint ensures that the shared codes generated by different modal combinations are similar through augmented training that randomly discards modalities.

[0011] Preferably, the suitability verification mechanism in S3 includes: A 3-layer fully connected validator network is constructed with the concatenated vectors of two latent spaces as input and the appropriateness score in the range (0,1) is output. The hidden layer is activated by LeakyReLU and the output layer is activated by Sigmoid. During the training phase, positive and negative sample pairs are constructed, and the binary cross-entropy loss is used to make the latent vector pairs of the same sample score high and the unrelated sample pairs score low. If the initial sampling score is <0.5 during the inference phase, gradient ascent optimization with a learning rate of 0.01 is used to iteratively adjust the private latent vector until the score is ≥0.5 or the number of iterations reaches 10.

[0012] Preferably, the phased sampling strategy in S3 is as follows: During the training phase, random Gaussian sampling is performed on both potential spaces, and KL divergence constraints are introduced to make the posterior distribution approximate the standard normal distribution. The weighting coefficients are set using the β-VAE strategy. During the inference phase, deterministic sampling is used to directly take the mean value for the shared latent space, while temperature-regulated Gaussian sampling is used for the private latent space. By adjusting the temperature coefficient τ, three generation modes—deterministic, standard, and highly diverse—are achieved to meet the generation needs of different scenarios.

[0013] Preferably, the calculation formula for the FiLM characteristic-level linear modulation mechanism in S4 is as follows: ; in, i =1, 2, 3, 4 correspond to four layers of emotional modulation. and They represent the first i Characteristics of emotion modulation before and after layering and Representing the hierarchy number respectively i Related, non-shared scale modulation parameters and offset modulation parameters, For the speaker's emotional characteristics in the video, For the first The layer uses an emotional conditional vector. This is element-wise multiplication.

[0014] Preferably, the four-layer hierarchical generation architecture in S4 is represented by the facial action unit AU as the intermediate, specifically as follows: First layer: Generate coarse-grained 3DMM parameters based on joint latent representation and first-layer sentiment conditional vector; The second layer generates an intermediate AU representation based on the joint latent representation, coarse-grained 3DMM parameters, and the second-layer sentiment conditional vector. The third layer: Based on the joint latent representation, AU intermediate representation and the third layer sentiment conditional vector, the coarse-grained 3DMM parameters are refined to generate refined 3DMM parameters; The fourth layer generates the listener's 25-dimensional emotional attributes based on joint latent representation, refined 3DMM parameters, AU intermediate representation, and fourth-layer emotional conditional vector.

[0015] Preferably, the method employs an end-to-end training approach, and the total loss function is a weighted combination of reconstruction loss, regularization loss, and constraint loss, specifically: ; in, For 3DMM reconstruction loss, Reconstructing loss for emotional attributes For head pose reconstruction loss, To share the potential space KL divergence loss, For the private latent space KL divergence loss, For potential spatial decoupling loss, For cross-modal consistency loss, For loss of emotional consistency, Loss due to suitability constraints For AU biomechanical constraint loss.

[0016] Therefore, the present invention employs the above-mentioned multimodal facial response generation method, and the technical effects are as follows: By constructing a trimodal input architecture of audio, video, and text, and using Mamba-SSM to achieve efficient long sequence modeling, the system relies on a two-layer decoupled latent space balance to generate correctness and diversity. Combined with a four-layer FiLM hierarchical emotion modulation and a six-way cross-modal attention mechanism, it solves the core defects of existing technologies such as single modality, insufficient long-term modeling ability, weak emotion regulation, and insufficient cross-modal interaction. It can generate realistic, natural, context-adaptive, emotionally consistent, and diverse facial responses, significantly improving model performance and scene adaptability. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multimodal facial response generation method according to the present invention; Figure 2 This is a technical roadmap for a multimodal facial response generation method according to the present invention; Figure 3 This is a flowchart of the hierarchical generation network in a multimodal facial response generation method of the present invention; Figure 4 This is a flowchart of the appropriateness verification mechanism in a multimodal facial response generation method of the present invention; Figure 5 This is a flowchart of a staged sampling strategy method in a multimodal facial response generation method of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] The following detailed description, in conjunction with the accompanying drawings and embodiments, illustrates the specific implementation of the multimodal facial response generation method in this application.

[0020] Figure 1 This is a flowchart of a multimodal facial response generation method provided in an embodiment of this application. Figure 2 This application provides a technical roadmap for a multimodal facial response generation method. Figure 3 This is a hierarchical generative network flowchart provided as an embodiment of this application. (In conjunction with...) Figure 1 , Figure 2 , Figure 3 As shown in the embodiments of this application, the multimodal facial response generation method includes the following steps: S1. Obtain speaker multimodal feature information and encode it separately. The multimodal feature information includes audio features, text features, video features and video emotion features. Align the audio features, text features and video features to the same time step length T in the time dimension. As a specific implementation method, this step is the input feature preparation stage. Audio, text, and video features are encoded separately using Mamba-SSM encoding blocks, aligning the three modal features to the same time step length T to eliminate temporal misalignment between modalities. Existing methods mainly use dual-modal input of audio and video, lacking textual semantic information. In dialogue scenarios, although psychological research shows that visual information accounts for 55%, auditory information for 38%, and linguistic content for only 7%, textual semantics plays a decisive role in specific contexts (such as ironic expressions, metaphor understanding, and sentiment polarity judgment). Therefore, this step achieves standardized temporal alignment of multimodal features, laying a unified temporal foundation for subsequent cross-modal fusion and avoiding semantic matching deviations caused by temporal misalignment.

[0021] S2. The encoded audio, text, and video features are sequentially subjected to temporal self-attention enhancement, six-way cross-modal attention enhancement, adaptive weight fusion, and post-processing of fused features to obtain deeply fused speaker fusion features. As a specific implementation method, this step is a deep fusion stage of multimodal information. First, it enhances the temporal features of a single modality through temporal self-attention, and then achieves trimodal full-dimensional information interaction through six-way bidirectional cross-attention. After adaptive fusion and post-processing, a unified fused feature is obtained. Thus, this step fully captures complementary information among the three modalities, compensates for the shortcomings of single-modal information, and can adaptively model the influence weights of different modalities on facial response generation at different time periods, generating a deep fused feature that combines temporal correlation and cross-modal semantics.

[0022] S3. Based on the fusion features, a two-layer latent space model with shared latent space and private latent space is constructed. The information of the two spaces is separated through multiple decoupling constraints. An appropriateness verification mechanism is introduced to verify and adaptively adjust the rationality of the combination of the two spaces. A phased sampling strategy is adopted to obtain the joint latent representation. As a specific implementation method, this step is the core of generating controllability and diversity modeling. A 256-dimensional shared space encodes general semantics, and a 128-dimensional private space encodes individual styles. Through multiple decoupling constraints, appropriateness verification, and phased sampling, a 384-dimensional joint latent representation is obtained. Thus, this step achieves the decoupling of semantic correctness and stylistic diversity, balancing the contextual adaptability and personalized expression of the generated results, and avoiding semantic drift and content homogenization.

[0023] S4. Based on joint latent representation and speaker video emotional features, a four-layer hierarchical generation architecture is constructed. In each layer, a FiLM feature-level linear modulation mechanism is introduced to guide emotional conditions, thereby generating coarse-grained 3DMM parameters, AU intermediate representation, refined 3DMM parameters and listener emotional attributes in sequence. As a specific implementation method, this step is the core generation stage of facial response. Guided by 25-dimensional speaker emotional features, it completes hierarchical generation through four layers of FiLM emotion modulation, following the order of coarse-grained 3DMM parameters, AU intermediate representation, refined 3DMM parameters, and listener emotional attributes. Therefore, this step achieves refined and controllable emotion generation from coarse to fine, ensuring a high degree of matching between facial movements and emotional semantics, significantly improving the naturalness and accuracy of facial expression details.

[0024] S5 outputs refined 3DMM facial parameters and listener emotional attributes to complete multimodal facial response generation. The refined 3DMM parameters are used to drive the 3D facial model to generate visual animations, and the listener emotional attributes are used for high-level emotional semantic analysis of facial responses.

[0025] As a specific implementation method, this step is the final output stage, outputting 58-dimensional refined 3DMM parameters with uniform temporal length and 25-dimensional listener emotional attributes. This can be directly adapted to 3D rendering and emotional semantic analysis scenarios, with optional post-processing filtering to further enhance naturalness. Therefore, this step outputs end-to-end, practically applicable results that can directly drive 3D animation generation without additional conversion, while also supporting high-level emotional semantic analysis requirements.

[0026] In summary, the embodiments of this application provide a multimodal facial response generation method. By using audio, text, and video three-modal temporal alignment encoding, six-way cross-modal deep fusion, two-layer decoupled latent space modeling, and a four-layer hierarchical generation architecture design with FiLM emotion modulation, it solves the core defects of existing technologies such as single modality, insufficient long temporal modeling, imbalance between generation correctness and diversity, and weak emotion regulation. It achieves context-adaptive, emotion-consistent, natural and realistic facial response generation with personalized diversity.

[0027] As one possible approach, this method extracts comprehensive multimodal feature information from the speaker, encompassing four core categories: audio, text, video basic features, and video emotional features. This provides standardized, high-dimensional feature input for subsequent multimodal fusion and emotion-guided modeling. Throughout the feature extraction process, the temporal dimensions of the three basic modalities (audio, text, and video) are aligned uniformly first, and the timestamps and sampling intervals of the three modalities are precisely calibrated, ultimately ensuring that all three modalities have the same time step length T. The accompanying multimodal feature encoder has the capability to model long sequences up to 750 time steps, fully covering standard dialogue segments of 30 seconds in length and 25fps video frame rate. This fully adapts to the duration requirements of daily dialogue interactions, avoiding information truncation or temporal disorder issues in long sequence feature modeling.

[0028] As one possible implementation, the feature extraction core uses the Mamba-SSM encoding block as the basic temporal modeling unit. This encoding block has a standardized structural design, with each layer containing a selective state space layer, a nonlinear activation function, a layer normalization module, and a residual connection module. The residual connection module directly fuses the original input features of the encoding block with the module output features through element-wise addition, effectively alleviating the gradient vanishing problem in the deep network modeling process, while retaining the core information of the original features and improving the modeling stability and representation ability of long-sequence temporal features.

[0029] As a specific implementation, the audio feature module takes the original audio waveform signal with a sampling rate of 16kHz as input and uses a dedicated audio encoder built based on Mamba-SSM coding blocks to complete the encoding. First, the one-dimensional audio waveform is converted into 80-dimensional Mel spectrum features through a Mel spectrum transform layer. Then, four layers of Mamba-SSM coding blocks conforming to the above standard structure are used to extract audio temporal features layer by layer. This supports audio sequence modeling of up to 750 frames (30 seconds @ 25fps), ultimately outputting T×512-dimensional audio features. Compared to the traditional Transformer model O(n... 2 The Mamba model achieves O(n) linear complexity for time-series modeling, significantly reducing the computational overhead of long audio sequences and improving modeling efficiency.

[0030] Text feature modules transcribe dialogue text sequences For input, a pre-trained Chinese RoBERTa model combined with Mamba-SSM encoding blocks is used to build a dedicated text encoder. First, the text sequence is converted into 768-dimensional word vectors through a token embedding layer. Then, a 4-layer Mamba-SSM encoding block is applied to capture the long-range semantic dependencies of the text sequence. Finally, a time alignment module aligns the temporal length of the text features to T frames, consistent with the audio and video features, outputting T×512-dimensional text features. This eliminates the temporal misalignment problem between modalities, ultimately outputting a 512-dimensional text feature representation. .

[0031] The video feature module uses a sequence of video frames featuring the speaker's face. For input, a dedicated video encoder is built using a ResNet-50 backbone network combined with Mamba-SSM coding blocks. First, 2048-dimensional facial spatial features are extracted from a single frame using ResNet-50. Then, four layers of Mamba-SSM coding blocks are used to perform temporal dynamic modeling of continuous frame sequences, mining the temporal correlation features of facial movements and expression changes. Finally, the features are mapped to a unified 512-dimensional dimension through a linear projection layer, outputting T×512-dimensional video features, representing... This enables dimensional matching with audio and text features.

[0032] The video emotion feature module is a dedicated emotion guidance branch, based on the speaker's facial video frame sequence. As input, a multi-branch parallel facial emotion extraction network is employed to mine multi-dimensional emotional representations. This network uses ResNet-50 as the spatial encoder to extract facial representation features, and simultaneously sets up three parallel branches: an emotion classification branch outputs the probability distribution of 8 basic emotion categories through a fully connected layer combined with a Softmax activation function; a valence-arousal regression branch outputs continuous emotional values ​​in the 2D range [-1,1] through a fully connected layer combined with a Tanh activation function; and a facial action unit (AU) detection branch outputs the intensity values ​​of 15 core facial action units through a fully connected layer combined with a Sigmoid activation function, covering AU1 (inner eyebrow lift), AU2 (outer eyebrow lift), AU4 (brow furrow), AU6 (cheek lift), AU7 (eyelid tightening), AU10 (upper lip lift), AU12 (corner of mouth lift), AU15 (corner of mouth pull down), AU17 (chin lift), AU20 (corner of mouth stretch), AU23 (lip tightening), AU25 (lip separation), AU26 (jaw drooping), AU43 (eye closure), and AU45 (blinking). The final output is a T×25-dimensional speaker video emotional feature. ,in This indicates that the first 8 dimensions represent the probability of the sentiment category. The 9th dimension represents the continuous valence value, reflecting the positive or negative value of facial expressions. It is used to regulate the positive or negative emotional tendency that generates facial expressions, such as a positive smile or a negative frown. The 10th dimension represents a continuous arousal value, indicating the degree of emotional activation or arousal. It is used to regulate the intensity and dynamism of facial responses. For example, high arousal corresponds to exaggerated facial expressions, while low arousal corresponds to weak responses. Dimensions 11-25 represent the intensity values ​​of 15 facial action units, which are fine-grained codes for facial muscle activity. This guides the generative network to achieve fine-grained motion control in various parts of the face, ensuring that facial expressions are generated in accordance with emotional categories while also exhibiting natural local muscle changes. By piecing together the video, a complete emotional profile of the speaker can be formed. , used for joint potential representation Multi-level, differentiated emotion modulation is achieved. Through this combination, the category, positive or negative tendency, dynamic level, and local details of facial responses can be controlled simultaneously to generate highly realistic, emotionally consistent, and diverse facial responses.

[0033] As one possible implementation, after extracting and temporally aligning the speaker's multimodal basic features, the audio, text, and video features are all normalized into a temporal feature matrix with a time length T and a feature dimension d=512, satisfying the following conditions: This module employs a unified format to perform deep fusion processing on time-aligned audio, text, and video features. It breaks down information barriers between single modalities, fully explores complementary cross-modal information, and strengthens temporal contextual relationships, ultimately generating multimodal deep fusion features that combine core single-modal information with cross-modal interaction information. This module relies on learnable query, key-value mapping matrices and a scaled dot product attention mechanism to sequentially complete single-modal temporal enhancement, cross-modal information interaction, adaptive weight fusion, and post-processing optimization, ensuring consistent feature dimensions and temporal alignment throughout the process.

[0034] Before cross-modal interaction, inter-frame temporal modeling is performed on the three types of modal features using independent temporal self-attention modules. This process uncovers the continuous time-step dependencies within a single modality, strengthens long-term temporal context representation, filters local noise, improves feature representation quality, and avoids invalid features interfering with subsequent cross-modal interaction. The corresponding calculation formula is as follows: Audio timing enhancement: ; Text temporal enhancement: ; Video temporal enhancement: ; This temporal attention mechanism follows a unified implementation logic, and is applicable to any single-modal input feature. First, through three sets of learnable linear mapping matrices The original features are projected into the attention computation space to generate a query vector Q, a key vector K, and a value vector V, as shown in the following formula: , , ; in, The internal mapping dimension of the attention matrix is ​​typically 512 or 64 × the number of attention heads. The number of rows in the mapping matrix matches the input feature dimension d, and the number of columns matches the attention mapping dimension. match.

[0035] Subsequently, a scaled dot product attention mechanism is used to complete the temporal modeling, and the calculation formula is as follows: ; In this process, Q, K, and V all originate from the same modal feature, focusing on modeling the dependency relationships between frames within a single modality. This effectively enhances the contextual feature representation ability of long-term dialogue segments, and the enhanced three types of features still retain [the desired characteristics]. Dimensional format.

[0036] To fully capture the complementary and correlated information of audio, text, and video modalities and avoid the problem of one-sided information from a single modality, this module designs a six-way bidirectional cross-attention mechanism. This mechanism enables bidirectional information guidance and enhancement between any two modalities, compensating for the information shortcomings of single-modal features. The core logic of cross-attention is as follows: a query vector Q is generated from the target modality features, and key vectors K and value vectors V are generated from the source modality features. Feature enhancement of the target modality from the source modality is achieved through scaling dot product attention. The general calculation formula is: ; in, For the target modal features to be enhanced, To provide auxiliary information, the source modal features maintain the same dimensionality as the output features. Unchanged. Based on the interaction relationships of the three modalities, a total of six targeted cross-attention enhancement branches were designed, as follows: Text-Audio Attention: Targeting temporally enhanced audio features and using text features as the source, its core function is to leverage text semantics to assist in audio prosody understanding, resolving the semantic ambiguity problem of pure audio features. The calculation formula is as follows: ; Text-video attention: This approach targets temporally enhanced video features and uses text features as the source. Its core function is to leverage textual semantics to assist in interpreting facial visual expressions, avoiding ambiguity in single-frame expressions, and matching visual features with textual semantics. The calculation formula is as follows: ; Audio-Text Attention: Targeting temporally enhanced text features and using audio features as the source, its core function is to leverage audio prosody to enhance the understanding of textual sentiment and semantics, compensating for the lack of emotional expression in plain text. The calculation formula is as follows: ; Audio-video attention: Targeting temporally enhanced video features and using audio features as the source, its core function is to capture the temporal correlation of facial expressions through audio prosody variations, thereby enhancing the emotional sensitivity of visual features. The calculation formula is: ; Video-Text Attention: Targeting temporally enhanced text features and using video features as the source, its core function is to refine the sentiment tendency of the text based on facial expressions, correcting sentiment biases in the text's semantics. The calculation formula is: ; Video-Audio Attention: Targeting temporally enhanced audio features and using video features as the source, its core function is to enhance the accuracy of emotional representation of audio features by using facial expressions to assist in the interpretation of audio emotions. The calculation formula is: ; After completing unimodal temporal enhancement and six-way cross-modal attention enhancement, the system enters the hierarchical weighted fusion stage. The original temporal enhancement features of each modality are weighted and superimposed with the cross-attention enhancement features of the other two modalities to generate a strengthened unimodal fusion feature. This precisely controls the proportion of cross-modal information contribution, balancing core unimodal information with auxiliary cross-modal information. The specific weighting rules are as follows: Audio enhancement feature fusion: It is obtained by weighting the temporal enhancement features of the audio itself, superimposed with the cross-enhancement features of text to audio and video to audio; Text Enhancement Feature Fusion: It is obtained by weighting the text's own temporal enhancement features and superimposing the cross-enhancement features from audio to text and video to text; Video Enhancement Feature Fusion: It is obtained by weighting the temporal enhancement features of the video itself, superimposed with the cross-enhancement features of text to video and audio to video.

[0037] in This is a cross-modal fusion weighting coefficient used to precisely control the contribution ratio of cross-modal information from different sources. It can be set to a fixed constant or a model learnable parameter, with a default value of 0.5 to ensure balanced integration of cross-modal information.

[0038] For the enhanced three modal features, two final fusion strategies adapted to different scenarios are provided, and the output features are all unified as follows: Dimensions: Fusion Strategy 1 is a pre-weighted feature concatenation and fusion: suitable for dialogue scenarios with clear modal importance and fixed scenes, where the weights satisfy normalization constraints. For example, in voice dialogue scenarios where audio information is the primary component, the default configuration is audio weighting. Text weight Video weight The weighted splicing formula is: The concatenated feature dimension is Learnable linear projection matrix The high-dimensional spliced ​​features are compressed to obtain the final dimension-normalized version. Multimodal fusion features .

[0039] Fusion Strategy 2 is a learnable adaptive attention fusion: suitable for scenarios with complex scenes and dynamically changing modal importance. It achieves adaptive weighted fusion by dynamically calculating the modal weights at each time step. The weights are optimized autonomously by the model, requiring no manual pre-setting. The first step is dynamic fusion weight calculation. For time step t, the three types of modal enhancement features are concatenated to construct intermediate features. Normalized weights are then generated through nonlinear transformation and the Softmax function. The calculation formula is as follows: ; in, And it satisfies the weight normalization constraint. The second step is time-step weighted fusion, with the following formula: ; The final output dimension is also 1. Adaptive multimodal fusion features .

[0040] To integrate redundant information within the fused features, enhance feature representation capabilities, and optimize model training stability, post-processing optimization is performed on the initial multimodal fused features. A two-layer fully connected network combined with residual connections and layer normalization is employed to avoid the gradient vanishing problem in deep networks while preserving the core information of the original fused features. First, a two-layer multilayer perceptron (MLP) is used to refine the fused features; the calculation formula is as follows: ; in, It is a non-linear activation function. These represent the weights and bias parameters of the fully connected layer, respectively. Then, a residual connection module is introduced to add the original fused features and the MLP output features element-wise. This is then standardized using a layer normalization module, ultimately yielding the optimized speaker deep fusion features. The calculation formula is as follows: ; Preserving output feature dimensions It can be directly used for modeling downstream tasks such as emotion guidance and dialogue recognition.

[0041] To address the core challenge of balancing semantic correctness and expressive diversity in generative dialogue models, a two-layer decoupled latent space is designed. This space independently encodes, samples, and constrains shared semantic information and private style information, while also incorporating a decoupling mechanism and an appropriateness verification system. Latent vectors are extracted based on multimodal fusion features and original modal features. Through multiple constraints, spatial decoupling is achieved, forming a complete latent space modeling and quality control process.

[0042] The core function of the shared latent space is to encode cross-modal general semantic information, ensuring the semantic appropriateness and content correctness of generated responses. It is a 256-dimensional continuous vector space, i.e. The internal latent vectors follow a standard multidimensional Gaussian distribution, satisfying: ; Where I is the identity matrix, representing the diagonal covariance structure of the multidimensional normal distribution.

[0043] Shared latent vectors based on optimized speaker deep fusion features Extraction begins by compressing the temporal fusion features into a global feature representation using an attention pooling layer. This process removes redundant temporal information and extracts the core semantic representation. The calculation formula is as follows: ; Obtain global features d=512; then, the mean and log-variance of the Gaussian distribution are estimated separately using two independent multilayer perceptrons to avoid the problem of negative variance, i.e.: ; ; Finally, sampling is completed using the reparameterization technique to achieve gradient backpropagation. The sampling formula is as follows: ; Among them, noise item It follows a standard normal distribution.

[0044] The core function of the private latent space is to encode speaker-specific style information, providing diversity and uniqueness for generated responses. It is a 128-dimensional continuous vector space. The latent vectors also follow a multidimensional Gaussian distribution, satisfying... It forms an independent distribution system with the shared space.

[0045] The private latent vector is not based on fused feature extraction, but rather on three types of original modal features. The features are obtained by fusing independently encoded features, fully preserving the unique style details of each modality. First, a dedicated private encoder is built for each modality to extract the private style features of each modality. The corresponding formula is: ; ; ; Among them, the single-modal private subspace dimension .

[0046] The proprietary modal encoder adopts a standardized four-layer structure, taking into account feature mapping, temporal modeling, and stability optimization: The first part is the input mapping layer, which projects the 512-dimensional input modal features to the proprietary subspace, as shown in the formula: ,in The first part is a learnable linear mapping matrix; the second part is a temporal modeling layer, which uses a selective state-space model (Mamba-SSM) to model the internal temporal dependencies of a single modality, and further optimizes the training process through residual connections. The formula is: Alternatively, it can be replaced with Transformer, Recurrent Neural Network, or Convolutional Temporal Network as needed; the third part is the nonlinear and normalization layer, which enhances feature representation through nonlinear activation functions such as GELU and ReLU, and improves training stability by combining layer normalization, as shown in the formula. ;in, The first part is a non-linear activation function; the fourth part is the output mapping layer, which generates the final private temporal features, as shown in the formula. .

[0047] After extracting the private features of a single modality, the private features of the three modalities are concatenated along the feature dimension to obtain: Dimensions after splicing Then, global private features are obtained through attention pooling: ; Subsequently, the mean and log-variance of the Gaussian distribution in the private space are estimated using a two-layer multilayer perceptron, i.e.: ; Finally, the private latent vector is obtained through reparameterized sampling, as shown in the formula: Among them, the noise item .

[0048] To ensure that the shared latent space and the private latent space strictly encode information of different dimensions and to prevent information overlap and spatial confusion, four types of complementary decoupling constraints are designed to force complete decoupling of semantic and style information and integrate them into a unified weighted loss to ensure decoupling effect and training stability.

[0049] The core objective of mutual information minimization constraints is to minimize the mutual information between two potential spaces, i.e. Mutual information estimation and optimization are achieved using an adversarial mutual information neural estimator (MINE), with the corresponding loss function being: ;in, The neural network estimate is driven by parameter θ, and mutual information suppression is achieved by minimizing this loss.

[0050] By introducing an adversarial discriminant mechanism, the decoupling effect of the latent space is enhanced. The core objective of the adversarial decoupling constraint is to prevent the discriminator from accurately predicting a vector in one latent space from a vector in another, thus completely severing the connection between the two. First, a decoupling discriminator is constructed. D And construct pseudo-sample pairs: generate pseudo-shared vectors using private latent vectors. Generating pseudo-private vectors using shared latent vectors Then, the adversarial losses for the shared space and the private space are calculated separately. The overall adversarial loss is the sum of the two, i.e.: ; Among them, shared space combats losses: ; Private space combats loss: ; Spatial decoupling is achieved through adversarial game theory.

[0051] To verify and enhance the coding specificity of each latent space, a reconstruction decoupling constraint is introduced, forcing each latent space to independently recover its corresponding original features. This ensures that the shared space recovers general fusion semantics, while the private space recovers individual style information. A shared decoder and a private decoder are built separately. The shared decoder uses... As input, reconstruct and fuse features, output dimension , To fuse feature dimensions, namely: ; Private decoder with As input, reconstruct the concatenated private features, and output the dimension: , Concatenate dimensions for private features, i.e.: ; Both types of decoders are composed of stacked linear mapping, nonlinear transformation, layer normalization, and residual connection units. The reconstruction loss is calculated using mean square error, and the overall reconstruction decoupling loss is: The first constraint ensures semantic fidelity in the shared space, while the second constraint ensures style fidelity in the private space.

[0052] The shared latent space encodes cross-modal general semantics and should therefore be unaffected by missing modalities, maintaining stable and consistent encoding results. Therefore, cross-modal consistency constraints are designed to ensure the stability and robustness of the shared space semantic encoding. During training, a modality dropout enhancement strategy is employed, randomly dropping one or two modalities with a probability of 0.3. The shared latent vectors generated by the complete modality input and the partial modality input are compared, forcing the minimization of their differences. Let the shared vector generated by the complete modality be... The shared vector generated by some modes is The cross-modal consistency loss is: ; The four types of decoupling constraints are integrated into a unified weighted loss function to achieve end-to-end optimization. The weighting formula is as follows: ; The weight parameter is set to By allocating appropriate weights, the optimization intensity of various constraints is balanced, ensuring both decoupling effectiveness and model convergence speed.

[0053] To avoid generating logically contradictory and semantically inconsistent facial reactions after combining two-layer latent spaces, a dedicated appropriateness verification mechanism is built to achieve latent vector matching degree evaluation, constraint optimization, and adaptive adjustment during the inference stage. This ensures the rationality of the generated content is controlled throughout the entire process from training to inference, guaranteeing that the combination of shared semantics and private style is suitable for the needs of dialogue scenarios.

[0054] Figure 4 A flowchart illustrating the suitability verification mechanism and method provided in this application's embodiments. (Combined with...) Figure 4 As shown, the suitability verification mechanism method provided in this application includes the following steps: S301: Construct a 3-layer fully connected validator network with two concatenated vectors from the potential space as input, and output the appropriateness score in the range (0,1). The hidden layer is activated by LeakyReLU and the output layer is activated by Sigmoid. As a specific implementation, the suitability verifier is a lightweight fully connected feedforward network, and the input is the concatenated joint latent vector. The network structure consists of three fully connected layers (256-dimensional shared vector + 128-dimensional private vector), with dimensions varying from 384 to 256 to 128 to 1. The hidden layers employ the LeakyReLU activation function to balance gradient propagation and non-linear expression, while the output layer uses the Sigmoid activation function to compress the output to the (0,1) interval, thus obtaining the appropriateness score. The expression is: ; in, The LeakyReLU activation function is used. This is the Sigmoid activation function.

[0055] The scoring criteria are clearly defined: a score ≥ 0.8 indicates that the overall performance is appropriate and the combination logic is reasonable; a score ≤ 0.5 < 0.8 indicates that the performance is basically appropriate and the generated result is acceptable; a score < 0.5 indicates that the performance is inappropriate and the latent vector combination needs to be adjusted.

[0056] S302: During the training phase, positive and negative sample pairs are constructed, and the binary cross-entropy loss is used to make the latent vector pairs of the same sample score high and the unrelated sample pairs score low. As a specific implementation method, the training phase constructs positive and negative sample pairs to optimize the validator and latent space encoder, thereby enhancing the matching degree between shared and private latent vectors for the same sample. Positive sample pairs are those corresponding to the same sample. Negative sample pairs are obtained by randomly shuffling private vectors within a batch or by randomly sampling irrelevant private vectors within the dataset. Construct; calculate the appropriateness scores for positive and negative sample pairs respectively. The loss function is constructed using binary cross-entropy, namely: ; By minimizing this loss, positive sample pairs receive high scores and negative sample pairs receive low scores, thus ensuring semantic consistency within the latent space.

[0057] S303: If the initial sampling score is <0.5 during the inference phase, gradient ascent optimization with a learning rate of 0.01 is used to iteratively adjust the private latent vector until the score is ≥0.5 or the number of iterations reaches 10.

[0058] As a specific implementation method, the inference phase adaptively optimizes the initial sampling results to avoid generating inappropriate content. First, the initial private latent vector is obtained through standard reparameterization. The input validator yields the initial score. If the initial score is less than 0.5, perform gradient ascent optimization on the private latent vector, and update the formula as follows: ; Among them, learning rate The stopping condition is a score ≥ 0.5 or the number of iterations k = 10; if the initial score meets the standard, the initial private latent vector is used directly without adjustment.

[0059] This embodiment of the application solves the core pain points of semantic drift and inappropriate generated content caused by combining two-layer fully connected appropriateness validators with a lightweight three-layer fully connected appropriateness validator network, combined with binary cross-entropy supervised training of positive and negative sample pairs during the training phase and a gradient ascent adaptive optimization mechanism triggered by threshold during the inference phase. While preserving the diversity of generation, it ensures the semantic adaptability and logical rationality of the generated facial reactions throughout the process, significantly improving the model's scene adaptability and generation quality.

[0060] Figure 5 This flowchart illustrates the phased sampling strategy method provided in this application. Different phased sampling strategies are designed for the different objectives of the training and inference phases. The training phase focuses on learning and fully exploring the latent space distribution, while the inference phase focuses on semantic correctness and controllable diversity, achieving precise adaptation to the objectives of different phases. Figure 5 As shown in the embodiments of this application, the staged sampling strategy method includes the following steps: S311: During the training phase, random Gaussian sampling is performed on both potential spaces, and KL divergence constraints are introduced to make the posterior distribution approximate the standard normal distribution. The weighting coefficients are set using the β-VAE strategy. The core objective of the training phase is to enable the model to fully learn the continuous data distribution of the latent space and uncover complete semantic and style feature patterns. Therefore, random Gaussian sampling is used for both layers of the latent space. The shared latent vectors follow... Private latent vectors follow By randomly sampling to cover the entire latent space, the model can avoid overfitting the training data.

[0061] Simultaneously, a KL divergence regularization constraint is introduced to force the learned posterior distribution to approximate the standard normal distribution N(0,1), thereby optimizing the latent spatial structure. The corresponding KL loss is: ; ; Using a β-VAE strategy to weight the KL terms, set Smaller weights can both regulate the latent spatial structure and avoid excessively inhibiting the reconstruction ability, thus balancing representation learning and distribution constraints.

[0062] S312: During the inference phase, deterministic sampling is used to directly take the mean for the shared latent space, while temperature-regulated Gaussian sampling is used for the private latent space. By adjusting the temperature coefficient τ, three generation modes—deterministic, standard, and highly diverse—are achieved to meet the generation needs of different scenarios.

[0063] During the inference phase, priority is given to ensuring the semantic correctness of the generated content while achieving controllable diversity. Therefore, a differentiated sampling method is adopted: deterministic sampling is used for the shared latent space, directly taking the distribution mean as the latent vector, i.e. To fix the core semantic information and avoid semantic deviation caused by random perturbations; the private latent space uses temperature-regulated Gaussian sampling to achieve controllable diversity, as shown in the formula: ; Where τ is a temperature coefficient that controls the intensity of diversity.

[0064] The temperature coefficient τ has a flexible adjustable range, corresponding to different generation modes: τ=0 represents a deterministic generation mode with zero variance and no style perturbation, suitable for scenarios with high accuracy requirements; τ=1 represents a standard generation mode, balancing stability and diversity; τ>1 represents a high-diversity mode, amplifying style perturbation, suitable for scenarios emphasizing expressive richness. Finally, the two-layer latent vectors are concatenated to obtain the joint latent representation: ; The score is evaluated through an appropriateness validator, and the joint representation can be directly input into the subsequent hierarchical generative network to achieve the step-by-step generation of the listener's facial response.

[0065] This embodiment of the application designs a differentiated phased sampling strategy for the training and inference stages. During the training stage, random Gaussian sampling is used for the two-layer latent space, combined with KL divergence constraints and β-VAE weighted optimization to optimize the latent space structure. During the inference stage, deterministic sampling is used for the shared space, and temperature-regulated Gaussian sampling is used for the private space, achieving multi-mode generation adaptation. This scheme solves the pain point of balancing distributed learning and generation controllability in the model. While ensuring the semantic correctness of the generated data, it achieves flexible and controllable generation diversity, significantly improving the model's scene adaptability.

[0066] Based on the dual-layer decoupled latent space and the 25-dimensional refined facial emotional features of the speaker extracted in step S1, this embodiment designs a four-layer progressive hierarchical generation architecture, coupled with a FiLM feature-level emotion modulation mechanism throughout the entire process, to achieve progressively refined generation from coarse-grained facial geometry to fine-grained facial movements, and then to high-level emotional semantics. The overall architecture follows the logic of "basic construction - fine-grained correction - semantic alignment". Each layer inherits the output features and emotional conditions of the upper layer. The FiLM mechanism realizes dynamic guidance of the speaker's emotions, and with the support of biomechanical, geometric and consistency constraints, it ensures that the generated listener facial responses conform to physiological laws and are highly adapted to the speaker's emotions. Finally, it outputs two core results: high-quality 3DMM facial parameters and high-level emotional attributes of the listener.

[0067] This module uses the FiLM (Feature-Level Linear Modulation) mechanism as its core emotion guidance unit, abandoning the weak correlation guidance method of traditional feature splicing. It generates exclusive affine transformation parameters based on the speaker's emotional conditions, and dynamically modulates the intermediate features of each layer element-wise, achieving deep integration of emotional information and generated features. This allows for refined emotion control layer by layer within a four-layer architecture. Its core feature-level affine transformation formula is as follows: ;in, i =1, 2, 3, 4 correspond to four layers of emotional modulation. and They represent the first i Features before and after layered emotion modulation and Representing the hierarchy number respectively i Related, non-shared scale modulation parameters and offset modulation parameters, This is element-wise multiplication. This represents the 25-dimensional speaker video emotion features extracted in step S1. For the first The layer uses an emotional condition vector.

[0068] Both scale and offset modulation parameters are generated from the speaker's emotion vector using a lightweight 25-128-d two-layer MLP network, where d represents the dimension of the corresponding feature to be modulated. Differential activation functions are set for the two types of parameters: scale modulation parameter... Activation is achieved using the Softplus function, with the output range stabilizing between 0 and 4, and a default center value of 1; offset modulation parameters... Activated using the Tanh function, the output range is controlled between -2 and 2, with a default center value of 0. The intensity of emotional guidance can be adjusted according to the needs of each layer of generated tasks, achieving precise guidance throughout the entire process.

[0069] First, the output results of the two-layer latent space are integrated and concatenated to obtain the joint latent representation: ; This vector integrates cross-modal general semantics with speaker-specific style information; it also incorporates 25-dimensional speaker video sentiment features. This serves as the core basis for guiding emotions across all levels.

[0070] The first layer is the basic construction layer for facial responses, and its core objective is based on joint latent representations. and the first layer of emotional condition vector The system generates coarse-grained 3DMM parameters, which is based on the joint latent representation of emotion category and valence pair to perform the first layer of emotion modulation. The coarse-grained 3DMM parameters of the listener are generated by the coarse-grained 3DMM decoder, and the basic geometric and posture framework of the listener's facial response is built to provide a stable foundation for subsequent fine-grained correction.

[0071] The first round of FiLM emotion modulation is adapted to the parameter range of the coarse-grained 3DMM generation task, generating exclusive modulation parameters through two layers of MLP: ; ; Among them, scale parameter The value range is (0.5, 3.5), and the offset parameter... The value range is (-1.5, 1.5), and the feature to be modulated is... The potential representation after the first layer of emotional guidance is obtained through affine transformation: This process injects the speaker's emotional information into core latent features, enabling the initial guidance of emotion on the basic facial structure.

[0072] To generate temporally coherent 3DMM parameters, the latent representation guided by the first layer of sentiment is... Input a coarse-grained 3DMM decoder and generate coarse-grained 3DMM parameters through the following steps.

[0073] The latent representation after single-frame modulation is expanded into a temporal feature sequence, which is then superimposed with a learnable positional encoding: ; Subsequently, a 3-layer Mamba-SSM decoding block is used to complete temporal modeling, uncovering the temporal dependencies of facial parameters. Finally, the model is mapped to a 58-dimensional 3DMM parameter space through a linear projection layer, outputting coarse-grained 3DMM parameters. The 58-dimensional parameters contain two core pieces of information: the first 52 dimensions are facial expression parameters based on the Basel Face Model (BFM) 3DMM expression base, which are responsible for controlling the basic deformation of facial muscles to achieve basic facial expressions such as smiling, frowning, and opening the eyes; the last 6 dimensions are head posture parameters, including 3-dimensional rotation angles (pitch, yaw, and roll) and 3-dimensional translation (displacement in the x, y, and z directions), which control the overall orientation and spatial position of the head.

[0074] The output of this layer consists of two parts: one is the coarse-grained 3DMM parameters. Second, the latent representation after the first layer modulation. .

[0075] The second layer is a fine-grained facial motion guidance layer, whose core objective is based on joint latent representation. Coarse-grained 3DMM parameters Second layer of emotional condition vector The generation of AU intermediate representation involves performing a second layer of emotion modulation on the joint latent representation and coarse-grained 3DMM parameters based on the intensity and arousal of facial action units. The AU intermediate representation decoder generates a 15-dimensional facial action unit (AU) intermediate representation, which accurately depicts the fine activation state of facial muscles, making up for the lack of detail in the coarse-grained 3DMM parameters. At the same time, biomechanical constraints ensure that the AU conforms to the movement law of human facial muscles.

[0076] This layer's input integrates core information from the upper layer, providing rich, multi-dimensional feature support for AU generation: joint latent representation. Coarse-grained 3DMM parameters These two types of information are copied and extended along the time dimension to T frames, ensuring that the two types of inputs are perfectly aligned in the time dimension. Then, they are concatenated and fused along the feature dimension to obtain the second layer of input features: ; The second round of sentiment feature modulation will be implemented using the same FiLM modulation logic: , The second layer of emotional guidance is obtained through affine transformation: This further enhances the ability of emotional information to guide fine-grained facial muscle movements.

[0077] The potential expression after guiding the second layer of emotion Input the AU intermediate representation decoder and generate the AU intermediate representation through the following steps.

[0078] First, a two-layer Mamba-SSM encoding block is used to capture the temporal dynamic features of the AU sequence. Then, an AU decoupling attention layer is constructed through an 8-head multi-head attention mechanism to explicitly model the correlation and dependency relationships of 15 AUs. Finally, the intermediate representation of the AU is output through a linear projection layer combined with a Sigmoid activation function. The 15 dimensions correspond to the core facial movement units, with values ​​ranging from [0,1], and the values ​​correspond to the muscle activation intensity.

[0079] To ensure the rationality of the generation, three types of biomechanical constraints are applied: mutual exclusion constraint, cooperative constraint, and temporal smoothing constraint of AU. The loss function is used to optimize and eliminate AU combinations that violate the logic of muscle movement.

[0080] The first type is mutually exclusive AU constraints, which are for AUs with opposite motion directions that cannot be activated at high intensity simultaneously. Typical examples are AU12 (corner of the mouth upturned) and AU15 (corner of the mouth downturned). The constraint loss calculation formula is as follows: The first type is punishment for situations where both are simultaneously highly activated; the second type is AU collaborative constraint, which targets physiologically coordinated AUs, typically such as the smiling combination of AU6 (cheek lifting) and AU12 (corner of the mouth turning up), through collaborative rewards. Both are encouraged to be activated simultaneously, and integrated into the overall loss in the form of negative loss. The loss calculation formula is as follows: The third type is the AU temporal smoothing constraint, which restricts drastic changes in AU activation intensity between adjacent frames. The loss calculation formula is as follows: This ensures that facial movements are smooth and natural.

[0081] The core output of this layer is a 15-dimensional AU intermediate representation. This is passed to the third layer, providing fine-grained guidance for refining 3DMM parameters.

[0082] The third layer is the facial parameter refinement layer, and its core objective is based on joint latent representation. AU represents and the third layer of emotional conditional vector The coarse-grained 3DMM parameters are refined to generate fine-grained 3DMM parameters. Specifically, the third-layer emotion modulation is performed on the joint latent representation, coarse-grained 3DMM parameters, and AU intermediate representation based on the facial action unit intensity, and the fine-grained 3DMM parameters are generated through the fine-grained 3DMM decoder.

[0083] This layer's input integrates three key types of information: joint latent representation. Coarse-grained 3DMM parameters AU represents the middle part The third layer of input features is obtained by concatenating along the feature dimensions: ; The third round of FiLM emotion modulation is implemented, and the parameter generation logic is consistent with the first two layers: , Features obtained after modulation The feature is then input into the refined 3DMM decoder guided by AU for targeted parameter correction.

[0084] The modulated features are input into the AU-guided refined 3DMM decoder, which first enhances key feature extraction through the AU attention enhancement module: ,in Temporal modeling is then performed using a three-layer Mamba-SSM fine-grained encoding block with an embedded AU attention enhancement mechanism to obtain the learnable AU-feature mapping matrix. This step adjusts the feature weights based on the AU; features are enhanced in frames with high AU activation and suppressed in frames with low AU activation, achieving precise control of AU over 3DMM refinement. Only detailed adjustments need to be learned, reducing the learning difficulty and ensuring continuity with coarse-grained parameters. Compared to direct generation, the AU-guided mechanism improves the ability to capture facial details, especially in the generation of micro-expressions (such as smiles and frowns). The residual refinement layer learns the corrected residuals of the coarse-grained 3DMM parameters. ; Finally, refined 3DMM parameters are obtained through residual superposition: ; in The learnable correction strength coefficient is initially set to 0.3 to balance the stability of the foundation structure with the correction effect.

[0085] Temporal smoothing is applied to the last 6-dimensional head pose parameters, and dual geometric constraints, namely hard constraints on parameter range and consistency constraints between expression and AU, are applied to ensure that the parameters conform to the physiological range and are highly matched with the intermediate representation of AU.

[0086] As a specific implementation method, additional temporal smoothing is applied to the last 6 dimensions of head pose parameters to avoid abrupt changes in head pose. The smoothing formula is as follows: ; Initial frame satisfies This further enhances the naturalness of head movements.

[0087] To ensure that the generated 3DMM parameters are within a reasonable physiological range, dual geometric constraints are applied: firstly, a hard constraint on the parameter range, limiting the facial expression parameters to a certain range. The first constraint is the setting of a compliant range for rotation angles: Pitch ∈ [-30°, +30°], Yaw ∈ [-45°, +45°], Roll ∈ [-20°, +20°], to prevent parameters from exceeding the limits of human movement; the second constraint is the expression-AU consistency constraint, achieved through a pre-trained 3DMM-AU mapping function. The consistency loss is calculated as follows: Ensure a high degree of match between the refined 3DMM facial expression parameters and the AU intermediate representation. The core output of this layer is the refined 3DMM parameters. The first 52 dimensions are refined facial expression parameters that fit the facial muscle movements, and the last 6 dimensions are smoothed head posture parameters. At the same time, the refined 3DMM parameters are passed to the fourth layer to complete the final emotional semantic alignment.

[0088] The fourth layer is the high-level sentiment semantic alignment layer, whose core objective is to align based on joint latent representations. Refined 3DMM parameters AU represents and the fourth layer of emotional conditional vector The system generates 25-dimensional emotional attributes for the listener. This involves performing fourth-layer emotional modulation on the joint latent representation, refined 3DMM parameters, and intermediate representations of facial action units based on emotional category, valence, and arousal. The listener's emotional attributes are then generated through an emotional attribute decoder. This layer generates 25-dimensional emotional attributes for the listener that are highly consistent with the underlying facial parameters, achieving bidirectional alignment between facial actions and emotional semantics, and ensuring the consistency between the listener's emotional expression and the speaker's emotions and their own facial actions.

[0089] This layer integrates all the core outputs from the first three layers, providing full-dimensional feature support for high-level emotion generation, and concatenates them to obtain the input features for the fourth layer: ; The final round of FiLM emotion modulation was performed, and the parameter generation logic followed the unified standard throughout the entire process. ; ; Features obtained after modulation It deeply integrates underlying facial features with emotional information to ensure the accuracy of emotional attribute generation.

[0090] The modulated features are input into the sentiment attribute decoder, which incorporates a two-layer Mamba-SSM sentiment coding block to model the dynamic changes in sentiment over time. Three parallel output heads are set up to synchronously generate multi-dimensional sentiment attributes: one is the sentiment classification head, which outputs the probability distribution of eight sentiment categories through a linear layer (457-2) combined with the Softmax activation function. The first is the valence-arousal regression head, which uses a linear layer (457-2) combined with the Tanh activation function to output a 2D continuous emotion value. Valence and Arousal both range from -1 to 1, quantifying nuanced emotional states; thirdly, the AU intensity prediction head, through a linear layer (457-15) combined with a Sigmoid activation function, outputs 15-dimensional AU intensity values. The value range is [0,1], which echoes the middle representation of the underlying AU.

[0091] The three types of results are combined to obtain the final 25-dimensional listener emotional attributes: ; To ensure a high degree of consistency between high-level emotional attributes and low-level facial parameters and speaker emotions, and to prevent contradictions between emotional semantics and facial movements, four types of weighted emotional consistency constraints are applied. The overall loss weights are reasonably allocated, taking into account the optimization strength of each constraint: The first type is the AU internal consistency loss, which constrains the consistency between the intermediate representation of the fourth-layer generated AU and the second-layer AU, allowing a reasonable tolerance of ±0.15, and the loss is: ; The second category is sentiment-AU semantic consistency loss, which is based on the FACS sentiment coding system theory. It matches sentiment categories with corresponding AU activation patterns, and the loss is: ; The third category is sentiment-3DMM consistency loss, which involves pre-training a 3DMM into a sentiment classifier. Pseudo-sentiment labels are generated, and consistency is constrained using KL divergence. The loss is: ; The fourth category is valence-arousal consistency loss, based on the emotional loop model, matching the theoretical correspondence between continuous and discrete emotions, with the loss being: ; The overall loss of affective consistency is: ; This layer is the final output layer of the entire hierarchical generation architecture, with the core output being 25-dimensional listener emotional attributes. Together with the refined 3DMM parameters output from the third layer, these constitute the two core results of the model, corresponding to the physical geometric representation of the listener's facial response and the high-level emotional semantic representation, respectively, thus fully realizing the task of generating listener facial responses based on the speaker's multimodal information.

[0092] The system's final output time sequence length is uniformly T frames. In a typical scenario, T=750, which corresponds to a standard dialogue segment with a duration of 30 seconds and a frame rate of 25fps, perfectly matching the time sequence specifications of the initial feature extraction.

[0093] The first output is the refined 3DMM parameters. The first parameter is the core physical parameter that drives the generation of visual dynamic expressions from 3D facial models. It can be directly input into the 3D rendering engine to generate facial reaction animations of listeners; the second output is a 25-dimensional listener emotional attribute. It is a high-level semantic representation of facial responses and can be used independently for dialogue sentiment analysis and interaction rationality assessment.

[0094] The system internally sets up a 15-dimensional AU intermediate representation. As a dedicated intermediate feature, it is only used for internal process transmission within the model and is not output externally. It undertakes three core functions: fine-grained guidance of 3DMM parameters, verification of emotional consistency, and transmission of facial motion information throughout the entire process.

[0095] To optimize application performance, this solution adds an optional post-processing workflow to suppress noise and abrupt changes, enhance naturalness, and can be enabled as needed: refined 3DMM parameters are smoothed using Savitzky-Golay filtering to eliminate high-frequency facial jitter; low-pass filtering is applied to the 6-dimensional head pose parameters to suppress abrupt motion changes; and the emotional probability distribution uses a time-series moving average to smooth emotional jumps and conform to real-world interaction logic.

[0096] This model constructs a multi-dimensional weighted total loss function that balances reconstruction accuracy, latent spatial structure, emotional consistency, and motion rationality. It consists of three main modules: reconstruction, regularization, and constraints, balancing various optimization objectives. The total loss satisfies the following: ; Reconstruction loss reduces the error between prediction and truth, ensuring output fidelity. (Weight) .

[0097] 3DMM Reconstruction Loss ; Emotional reconstruction loss integrates classification, regression, and AU prediction, with weights : ; attitude reconstruction loss Weight This enhances the posture optimization effect.

[0098] Regularization loss regularizes the latent space, prevents overfitting, and ensures decoupling: shared vs. private latent space KL loss , A uniformly configured minimum weight of 0.0002 is applied. Decoupling loss fusion with multiple constraints: weights This forces a complete decoupling of shared semantics and private style, eliminating information interference.

[0099] ; Cross-modal consistency loss Independent weights Overall Emotional Consistency Loss: Weighting ; ; Appropriateness constraint loss: weight ; ; AU biomechanical constraint loss: weights It conforms to the laws of physiological movement. ; The final total loss function is obtained by summing all the sub-losses according to their corresponding weights, as shown in the following formula: ; The loss function design has undergone fine-tuning of weights to achieve an optimal balance between the accuracy of the generated result reconstruction, the structuring of the latent space, the consistency of emotional semantics, and the biomechanical rationality of facial movements. This ensures that the model converges quickly and outputs high-quality, highly reasonable listener facial response results, making it suitable for various human-computer dialogue and virtual interaction scenarios.

[0100] Therefore, this invention employs the aforementioned multimodal facial response generation method. For the task of generating listener facial responses, it first extracts aligned temporal features and refined 25-dimensional emotional features from the speaker's audio, text, and video modalities, and then completes deep feature fusion through a multimodal attention mechanism. It constructs a 256-dimensional shared and 128-dimensional private two-layer decoupled latent space, relying on four types of decoupling rules such as mutual information minimization and adversarial constraints to separate general semantics and personalized style information. An appropriateness verification mechanism ensures the rationality of latent vector combinations, and a phased sampling strategy balances the distribution of the training phase and the controllable generation during the inference phase. The entire process utilizes the FiLM feature-level linear modulation mechanism to achieve... The speaker's emotions are guided layer by layer, and a four-layer hierarchical generation architecture is built. From coarse-grained 3DMM facial parameter generation and extraction of intermediate representations of 15-dimensional AU facial motion units, to fine-grained 3DMM parameter correction, the final output is 25-dimensional high-level emotional attributes of the listener. At the same time, it is equipped with multiple constraints such as AU biomechanics, emotional consistency, and 3DMM geometry. Combined with a weighted and integrated complete loss function, it ensures that the generated facial response is semantically correct, the posture is natural, and conforms to the laws of physiological movement, while taking into account individual diversity. Finally, it outputs two core results: fine-grained 3DMM driving parameters and high-level emotional attributes with regular temporal order. It can be directly used for 3D facial animation rendering and emotional interaction analysis.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating multimodal facial responses, characterized in that, Includes the following steps: S1. Obtain speaker multimodal feature information and encode it separately. The multimodal feature information includes audio features, text features, and video features. Align the audio features, text features, and video features to the same time step length T in the time dimension. S2. The encoded audio, text, and video features are sequentially subjected to temporal self-attention enhancement, six-way cross-modal attention enhancement, adaptive weight fusion, and post-processing of fused features to obtain deeply fused speaker fusion features. S3. Based on the fusion features, a two-layer latent space model with shared latent space and private latent space is constructed. The information of the two spaces is separated through multiple decoupling constraints. An appropriateness verification mechanism is introduced to verify and adaptively adjust the rationality of the combination of the two spaces. A phased sampling strategy is adopted to obtain the joint latent representation. S4. Based on joint latent representation and speaker video emotional features, a four-layer hierarchical generation architecture is constructed. In each layer, a FiLM feature-level linear modulation mechanism is introduced to guide emotional conditions, thereby generating coarse-grained 3DMM parameters, AU intermediate representation, refined 3DMM parameters and listener emotional attributes in sequence. S5 outputs refined 3DMM facial parameters and listener emotional attributes to complete multimodal facial response generation. The refined 3DMM parameters are used to drive the 3D facial model to generate visual animations, and the listener emotional attributes are used for high-level emotional semantic analysis of facial responses.

2. The multimodal facial response generation method according to claim 1, characterized in that, S1 uses Mamba-SSM coding blocks to complete the temporal modeling of audio, text, and video features. The Mamba-SSM coding block includes a selective state space layer, nonlinear activation, layer normalization, and residual connections. The residual connections use an element-wise addition method to fuse the input features and the encoded output.

3. The multimodal facial response generation method according to claim 1, characterized in that, The six-way cross-modal attention enhancement in S2 constructs a bidirectional cross-attention mechanism for text-audio, text-video, audio-text, audio-video, video-text, and video-audio. It achieves cross-modal information interaction through learnable query, key, and value mapping matrices, and enhances the feature representation of another modality by utilizing semantic / prosodic / visual information from one modality, thus fully capturing complementary information between the three modalities.

4. The multimodal facial response generation method according to claim 1, characterized in that, In the S3 two-layer latent space model, the shared latent space encodes semantic information shared between modalities to ensure the correctness and appropriateness of generated responses, while the private latent space encodes individual-specific style information to provide diversity and personalization of generated responses. The shared latent space is obtained by reparameterizing the fused features after attention pooling and MLP estimation of mean and variance. The private latent space is obtained by reparameterizing the original features of audio, text, and video after encoding and concatenation by a dedicated encoder, pooling, and MLP estimation of mean and variance. The proprietary modal encoder includes an input mapping layer, a selective state-space modeling layer with residual connections, a nonlinearity and normalization layer, and an output mapping layer.

5. The multimodal facial response generation method according to claim 4, characterized in that, S3's multiple decoupling constraints include mutual information minimization constraints, adversarial decoupling constraints, reconstruction decoupling constraints, and cross-modal consistency constraints. The comprehensive decoupling loss is: ； in, To minimize the constraint loss of mutual information, In order to combat losses as a whole, To reconstruct the decoding loss, For cross-modal consistency loss, All are decoupling constraint weights; The mutual information minimization constraint uses MINE to estimate and minimize the mutual information between two potential spaces. The adversarial decoupling constraint introduces a discriminator that prevents it from predicting one space from another. The reconstruction decoupling constraint enables the two spaces to independently recover their corresponding information. The cross-modal consistency constraint ensures that the shared codes generated by different modal combinations are similar through augmented training that randomly discards modalities.

6. The multimodal facial response generation method according to claim 4, characterized in that, The suitability verification mechanism in S3 includes: A 3-layer fully connected validator network is constructed with the concatenated vectors of two latent spaces as input and the appropriateness score in the range (0,1) is output. The hidden layer is activated by LeakyReLU and the output layer is activated by Sigmoid. During the training phase, positive and negative sample pairs are constructed, and the binary cross-entropy loss is used to make the latent vector pairs of the same sample score high and the unrelated sample pairs score low. If the initial sampling score is <0.5 during the inference phase, gradient ascent optimization with a learning rate of 0.01 is used to iteratively adjust the private latent vector until the score is ≥0.5 or the number of iterations reaches 10.

7. The multimodal facial response generation method according to claim 4, characterized in that, The phased sampling strategy in S3 is as follows: During the training phase, random Gaussian sampling is performed on both potential spaces, and KL divergence constraints are introduced to make the posterior distribution approximate the standard normal distribution. The weighting coefficients are set using the β-VAE strategy. During the inference phase, deterministic sampling is used to directly take the mean value for the shared latent space, while temperature-regulated Gaussian sampling is used for the private latent space. By adjusting the temperature coefficient τ, three generation modes—deterministic, standard, and highly diverse—are achieved to meet the generation needs of different scenarios.

8. The multimodal facial response generation method according to claim 1, characterized in that, The calculation formula for the FiLM characteristic-level linear modulation mechanism in S4 is as follows: ； in, i =1, 2, 3, 4 correspond to four layers of emotional modulation. and They represent the first i Characteristics of emotion modulation before and after layering and Representing the hierarchy number respectively i Related, non-shared scale modulation parameters and offset modulation parameters, For the speaker's emotional characteristics in the video, For the first The layer uses an emotional conditional vector. This is element-wise multiplication.

9. The multimodal facial response generation method according to claim 1, characterized in that, The four-layer hierarchical generation architecture in S4 is represented by the facial motion unit (AU) as the intermediate layer, specifically as follows: First layer: Generate coarse-grained 3DMM parameters based on joint latent representation and first-layer sentiment conditional vector; The second layer generates an intermediate AU representation based on the joint latent representation, coarse-grained 3DMM parameters, and the second-layer sentiment conditional vector. The third layer: Based on the joint latent representation, AU intermediate representation and the third layer sentiment conditional vector, the coarse-grained 3DMM parameters are refined to generate refined 3DMM parameters; The fourth layer generates the listener's 25-dimensional emotional attributes based on joint latent representation, refined 3DMM parameters, AU intermediate representation, and fourth-layer emotional conditional vector.

10. The multimodal facial response generation method according to claim 1, characterized in that, The method employs an end-to-end training approach, with the total loss function being a weighted combination of reconstruction loss, regularization loss, and constraint loss, specifically: ； in, For 3DMM reconstruction loss, Reconstructing loss for emotional attributes For head pose reconstruction loss, To share the potential space KL divergence loss, For the private latent space KL divergence loss, For potential spatial decoupling loss, For cross-modal consistency loss, For loss of emotional consistency, Loss due to suitability constraints For AU biomechanical constraint loss.

Images