Digital human emotion simulation method and system based on multi-modal data fusion

Abstract

The application relates to a digital human emotion simulation method and system based on multi-modal data fusion, which comprises obtaining driving text, voice signals, self-defined emotion intensity and personality style parameters; text emotion features, voice content features and voice emotion features are respectively extracted through a pre-trained model, and multi-modal emotion features are obtained through fusion; a content encoder and an emotion encoder are used to decouple the voice content features and the multi-modal emotion features, and a cross reconstruction strategy is used to make the content features and the emotion features independent of each other; the emotion intensity and the personality style parameters are converted into explicit control features through an embedding layer, and are spliced with the decoupled content features and emotion features to obtain fusion features; the fusion features are input into a Transformer decoder containing a biased multi-head self-attention and a biased cross-modal attention mechanism, a facial feature coefficient sequence is generated and is mapped to a digital human model, and the digital human model is driven to present expressions synchronized with the voice.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to a digital human emotion simulation method and system based on multimodal data fusion. Background Technology

[0002] With the rapid development of artificial intelligence and computer graphics technologies, digital humans have become a research hotspot in the field of human-computer interaction. From early simple voice assistants to today's virtual avatars with lifelike appearances, digital humans are gradually penetrating various industries such as entertainment, education, customer service, and healthcare.

[0003] Currently, mainstream digital human facial animation technologies primarily employ speech-based single-modal approaches. These methods use speech signals as the sole input and learn the mapping relationship between phonemes and lip movements through deep learning models to generate facial animations synchronized with speech. Representative works include FaceFormer and EmoTalk, which utilize temporal modeling architectures such as Transformer to achieve high accuracy in speech-lip-sync. Some improved methods further attempt to implicitly model emotional information from speech or control the intensity and category of generated expressions through explicit emotional tags. These methods typically output 3D mesh vertex coordinates or facial blending shape coefficients and can be applied to different digital human models through retargeting techniques.

[0004] The above methods have insufficient ability to model emotions based on single-modal inputs. Relying solely on speech signals makes it difficult to fully capture the emotional state during interactions. The content information and emotional features in the speech are coupled and interfere with each other, resulting in insufficient matching between facial expressions and context.

[0005] Based on this, this application provides a digital human emotion simulation method and system based on multimodal data fusion. Summary of the Invention

[0006] To address the shortcomings of single-modal input in emotion modeling, the difficulty in fully capturing emotional states during interaction by relying solely on speech signals, and the coupling and interference between content information and emotional features in speech, which leads to insufficient matching between facial expressions and context, this application provides a digital human emotion simulation method and system based on multimodal data fusion.

[0007] Firstly, this application provides a digital human emotion simulation method based on multimodal data fusion, which adopts the following technical solution: including:

[0008] The process involves acquiring multimodal input data, including driving text, corresponding speech signals, and user-defined emotion intensity and personality style parameters; extracting text emotion features from the driving text using a pre-trained text emotion feature extraction model; extracting speech content features from the speech signals using a pre-trained speech content feature extraction model; extracting speech emotion features from the speech signals using a pre-trained speech emotion feature extraction model; and fusing the text emotion features and the speech emotion features at the feature level to obtain multimodal emotion features.

[0009] The speech content features are input into the content encoder for encoding to obtain decoupled content features; the multimodal emotion features are input into the emotion encoder for encoding to obtain decoupled emotion features. The content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other.

[0010] The emotional intensity parameter and personality style parameter are respectively converted into continuous dense vectors through an embedding layer, and the two are concatenated to obtain the explicit control feature; the decoupled content feature, emotional feature and the explicit control feature are concatenated to obtain the fused feature;

[0011] The fused features are input into a Transformer-based emotion-driven decoder, which includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, resulting in a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

[0012] Preferably, the step of inputting the speech content features into a content encoder for encoding to obtain decoupled content features; and inputting the multimodal emotion features into an emotion encoder for encoding to obtain decoupled emotion features, wherein the content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other, including:

[0013] The speech content features are input into the content encoder, which consists of a linear layer and a 6-layer Transformer encoder, reducing the speech content features from 1024 dimensions to 512 dimensions and outputting the decoupled content features.

[0014] The multimodal emotion features are input into the emotion encoder, which consists of two linear layers and a ReLU activation layer, reducing the multimodal emotion features from 832 dimensions to 256 dimensions and outputting the decoupled emotion features.

[0015] A cross-reconstruction strategy is employed to jointly train the content encoder and the emotion encoder. Two pairs of samples with different contents and emotions are selected from the training data, and their content features and emotion features are extracted respectively. The content features of the first sample are cross-combined with the emotion features of the second sample and input into the decoder to generate the first predicted facial animation. The content features of the second sample are cross-combined with the emotion features of the first sample and input into the decoder to generate the second predicted facial animation. With real facial animation as supervision, the cross-reconstruction loss function is minimized to force the content features and emotion features to be independent of each other in the feature space, making the content features and emotion features orthogonal or approximately orthogonal in the feature space.

[0016] Preferably, the emotional intensity parameter and personality style parameter are respectively converted into continuous dense vectors through an embedding layer, and the two are concatenated to obtain explicit control features; the decoupled content features, emotional features, and explicit control features are concatenated to obtain fused features, including:

[0017] The emotion intensity parameter is input into the first embedding layer. The emotion intensity parameter is a two-dimensional one-hot encoded vector, which represents two discrete categories: high-intensity emotion and low-intensity emotion. The first embedding layer maps it into a 32-dimensional continuous dense vector to obtain the emotion intensity embedding feature.

[0018] The personality style parameters are input into the second embedding layer. The personality style parameters are 24-dimensional one-hot encoded vectors, which represent 24 different personality style categories. The second embedding layer maps them into 32-dimensional continuous dense vectors to obtain the personality style embedding features.

[0019] The emotional intensity embedding feature and the personality style embedding feature are concatenated into vectors to obtain the 64-dimensional explicit control feature. During training, the explicit control feature continuously optimizes the embedding matrix parameters through backpropagation, so that semantically similar categories are closer in the embedding space.

[0020] The decoupled content features, emotion features, and explicit control features are concatenated. The content features are 512-dimensional, the emotion features are 256-dimensional, and the explicit control features are 64-dimensional. The concatenated features are 832-dimensional. The concatenated features include fine-grained temporal content information of speech, global emotion information, and user-adjustable emotional intensity and personality style information.

[0021] Preferably, the fused features are input into a Transformer-based emotion-driven decoder, which includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long-term temporal dependencies and achieve modal alignment, resulting in a sequence of facial feature coefficients corresponding to each time step, including:

[0022] A Transformer-based emotion-driven decoder is constructed, which consists of multiple stacked decoding layers. Each decoding layer contains a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer adopts residual connections and layer normalization.

[0023] Periodic position coding is added to the fused features to obtain coded features carrying position information. The periodic position coding is generated using sine and cosine functions, with its period parameter set to sixteen. By taking the position index modulo the period and calculating the trigonometric function value, the model can perceive the motion pattern between frames and reduce lip-sync drift in long sequences.

[0024] The encoded features carrying location information are input into the biased multi-head self-attention sub-layer of the first decoding layer. In the biased multi-head self-attention sub-layer, the ALiBi mechanism is adopted to add a bias term that is linearly related to the distance between the query and the key to the scaled dot product attention score. The bias slope is set to the reciprocal of the head index plus one according to the attention head index, so that different attention heads punish distant positions at different rates to obtain self-attention output.

[0025] The self-attention output is residually connected to the input of the biased multi-head self-attention sub-layer and then layer normalized to obtain the first sub-layer output.

[0026] The output of the first sub-layer is used as a query, and the emotional feature sequence is used as the key and value, which are then input into the biased cross-modal multi-head attention sub-layer. At the same time, a memory mask matrix is ​​constructed. The memory mask matrix takes a value of zero only at the emotional feature positions within the current time step and a preset time window, and a value of negative infinity at other positions. The memory mask matrix is ​​introduced into the attention calculation to ensure that the attention mechanism only focuses on temporally aligned emotional features, thus obtaining the cross-modal attention output.

[0027] The cross-modal attention output is residually connected to the first sub-layer output and then layer-normalized to obtain the second sub-layer output; the second sub-layer output is input into the feedforward neural network sub-layer for nonlinear transformation to obtain the feedforward output; the feedforward output is residually connected to the second sub-layer output and then layer-normalized to obtain the final output of the current decoding layer.

[0028] The final output of the current decoding layer is used as the input of the next decoding layer. The decoding steps are repeated until the last decoding layer outputs the final feature sequence.

[0029] The feature sequence output from the last decoding layer is input into the fully connected mapping layer. The fully connected mapping layer reduces the feature dimension from 832 to 52 and constrains the output value between zero and one through the Sigmoid activation function to obtain the facial feature coefficient sequence corresponding to the time step.

[0030] Preferably, the output of the first sub-layer is used as a query, and the emotional feature sequence is used as the key and value, which are then input into the biased cross-modal multi-head attention sub-layer. Simultaneously, a memory mask matrix is ​​constructed, where the memory mask matrix takes a value of zero only at the emotional feature positions within the current time step and a preset time window, and a value of negative infinity at other positions. The memory mask matrix is ​​then introduced into the attention calculation to ensure that the attention mechanism only focuses on temporally aligned emotional features, resulting in cross-modal attention output, including:

[0031] The output of the first sub-layer obtained after residual connection and layer normalization is used as the query input. Each row of the query input corresponds to the decoding feature of a time step. The decoupled emotion feature sequence is used as the key input and value input. The frame rate of the emotion feature sequence is consistent with the frame rate of the facial animation. Each frame corresponds to an emotion feature vector.

[0032] Construct a memory mask matrix, where the number of rows in the memory mask matrix equals the number of time steps of the query input, and the number of columns equals the number of time steps of the emotion feature sequence. For each element in the memory mask matrix, if the column index is greater than or equal to the preset window scaling factor multiplied by the row index, and less than the preset window scaling factor multiplied by the row index plus one, then the corresponding element takes the value of zero; otherwise, it takes the value of negative infinity. The preset window scaling factor represents the number of emotion feature frames contained in the time window corresponding to each facial animation frame.

[0033] The query input is linearly projected onto multiple attention heads to obtain the query submatrix for each head; the key input and value input are linearly projected onto the same number of attention heads to obtain the key submatrix and value submatrix for each head.

[0034] For each attention head, calculate the product of the query submatrix and the transpose of the key submatrix, and then divide by the square root of the dimension of the key matrix to obtain the scaled dot product attention fractional submatrix; add the memory mask matrix to the scaled dot product attention fractional submatrix to obtain the biased attention fractional submatrix.

[0035] The biased attention score submatrix is ​​normalized row by row to obtain the attention weight submatrix; the attention weight submatrix is ​​multiplied by the value submatrix to obtain the attention output of each head;

[0036] All the attention outputs are concatenated and then fused through a linear projection layer to obtain the cross-modal attention output, which serves as the calculation result of this sub-layer.

[0037] Preferably, the step of adding periodic positional encoding to the fused features to obtain encoded features carrying positional information includes:

[0038] The fused features are used as the input feature sequence, which has a dimension of 832 and a time step of T. Each frame corresponds to a fused feature vector.

[0039] Construct a periodic positional encoding matrix with the same dimension as the input feature sequence, consisting of T rows and 832 columns. For the positional encoding values ​​at the pos-th time step, the 2i-th dimension, and the (2i+1)-th dimension, the encoding value of the 2i-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the sine function value of the independent variable. Similarly, the encoding value of the (2i+1)-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the cosine function value of the independent variable.

[0040] The periodic position encoding matrix is ​​added element by element to the input feature sequence to obtain the encoded features carrying position information.

[0041] Preferably, a multimodal emotion-driven model is constructed and trained; the multimodal emotion-driven model includes a feature extraction module, a feature decoupling module, an explicit control embedding module, and an emotion-driven decoder; the feature extraction module includes: a text emotion feature extraction model, a speech content feature extraction model, and a speech emotion feature extraction model; the feature decoupling module includes: a content encoder and an emotion encoder; and the explicit control embedding module is used to generate explicit control features.

[0042] The training specifically includes the following steps:

[0043] Construct a training dataset containing multiple sample pairs. Each sample pair includes driving text, speech signal, corresponding real facial feature coefficient sequence, and emotion category label. The training dataset contains various combinations of samples with the same semantic content but different emotion categories, as well as samples with the same emotion category but different semantic content.

[0044] The feature extraction module is initialized. The text sentiment feature extraction model adopts the DistilRoBERTa model pre-trained on the sentiment dataset, the speech content feature extraction model adopts the WavLM model pre-trained on a large unsupervised speech dataset, and the speech emotion feature extraction model adopts the EDWavLM model fine-tuned on the sentiment dataset. During the training process, the parameters of the feature extraction module are fixed and it is only used as a feature extractor.

[0045] The initialization feature decoupling module consists of a content encoder composed of a linear layer and a six-layer Transformer encoder, which reduces the speech content features from 1,024 dimensions to 512 dimensions; the emotion encoder consists of two linear layers and a ReLU activation layer, which reduces the multimodal emotion features from 832 dimensions to 256 dimensions.

[0046] The explicit control embedding module is initialized, which includes a first embedding layer and a second embedding layer. The first embedding layer maps two-dimensional emotion intensity one-hot encoding to thirty-two-dimensional emotion intensity embedding features. The second embedding layer maps twenty-four-dimensional personality style one-hot encoding to thirty-two-dimensional personality style embedding features.

[0047] The emotion-driven decoder is initialized, which consists of multiple stacked decoding layers and fully connected mapping layers. Each decoding layer contains a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer uses residual connections and layer normalization. The fully connected mapping layer maps the 832-dimensional features output by the decoder to 52-dimensional facial feature coefficients.

[0048] First and second samples are randomly selected from the training dataset. The first sample contains first semantic content, first emotion category and its corresponding first real facial feature coefficient sequence, and the second sample contains second semantic content, second emotion category and its corresponding second real facial feature coefficient sequence.

[0049] The driving text and speech signals of the first and second samples are input into the feature extraction module to extract the text emotion features, speech content features, and speech emotion features of the first sample, as well as the text emotion features, speech content features, and speech emotion features of the second sample. The text emotion features and speech emotion features of the first sample are fused to obtain the first multimodal emotion feature, and the text emotion features and speech emotion features of the second sample are fused to obtain the second multimodal emotion feature.

[0050] The speech content features of the first sample are input into the content encoder to obtain the first content feature; the first multimodal emotion feature of the first sample is input into the emotion encoder to obtain the first emotion feature; the speech content features of the second sample are input into the content encoder to obtain the second content feature; the second multimodal emotion feature of the second sample is input into the emotion encoder to obtain the second emotion feature.

[0051] The first content feature and the second emotion feature are concatenated and input into the emotion-driven decoder to generate the first predicted facial feature coefficient sequence; the second content feature and the first emotion feature are concatenated and input into the emotion-driven decoder to generate the second predicted facial feature coefficient sequence.

[0052] Obtain the first true facial feature coefficient sequence corresponding to the first semantic content and the second emotion category from the training dataset, and the second true facial feature coefficient sequence corresponding to the second semantic content and the first emotion category.

[0053] Calculate the joint loss function value, which is a weighted sum of the cross-reconstruction loss, self-reconstruction loss, velocity loss, and classification loss;

[0054] The cross-reconstruction loss is the sum of the squared Euclidean distance between the first predicted facial feature coefficient sequence and the first true facial feature coefficient sequence, plus the squared Euclidean distance between the second predicted facial feature coefficient sequence and the second true facial feature coefficient sequence.

[0055] The self-reconstruction loss is the sum of the squared Euclidean distance between the predicted sequence obtained by concatenating the first content feature and the first emotion feature and inputting it into the emotion-driven decoder and the first real facial feature coefficient sequence, plus the squared Euclidean distance between the predicted sequence obtained by concatenating the second content feature and the second emotion feature and inputting it into the emotion-driven decoder and the second real facial mixed shape coefficient sequence.

[0056] The velocity loss is the squared Euclidean distance between the difference vectors between adjacent frames in the predicted sequence and the difference vectors between adjacent frames in the true sequence.

[0057] The first and second emotion features are input into the classifier respectively to obtain the probability distribution of the predicted emotion category. The sum of the cross-entropy loss between the predicted emotion category and the actual emotion category label is calculated to obtain the classification loss.

[0058] Based on the joint loss function value, the parameters of the content encoder, emotion encoder, explicit control embedding module, and emotion-driven decoder are updated using the backpropagation algorithm.

[0059] Repeat the training steps until the joint loss function value converges or the preset number of training rounds is reached to obtain a trained multimodal emotion-driven model.

[0060] Secondly, this application discloses a digital human emotion simulation device based on multimodal data fusion, which adopts the following technical solution, including:

[0061] The feature extraction module is used to acquire multimodal input data, which includes driving text, speech signals corresponding to the driving text, and user-defined emotion intensity parameters and personality style parameters. It extracts text emotion features from the driving text using a pre-trained text emotion feature extraction model; extracts speech content features from the speech signals using a pre-trained speech content feature extraction model; extracts speech emotion features from the speech signals using a pre-trained speech emotion feature extraction model; and performs feature-level fusion of the text emotion features and the speech emotion features to obtain multimodal emotion features.

[0062] The feature encoding module is used to input the speech content features into the content encoder for encoding to obtain decoupled content features; and to input the multimodal emotion features into the emotion encoder for encoding to obtain decoupled emotion features. The content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other.

[0063] The feature fusion module is used to convert the emotion intensity parameter and the personality style parameter into continuous dense vectors through the embedding layer, and then concatenate the two to obtain the explicit control feature; the decoupled content feature, emotion feature and the explicit control feature are concatenated to obtain the fused feature;

[0064] The emotion-driven module is used to input the fused features into a Transformer-based emotion-driven decoder. The emotion-driven decoder includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, thereby obtaining a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

[0065] Thirdly, this application also provides a control device, the device comprising:

[0066] It includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor, such as the digital human emotion simulation method based on multimodal data fusion described above.

[0067] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the digital human emotion simulation method based on multimodal data fusion.

[0068] In summary, this application first obtains the driving text, the corresponding speech signal, and user-defined emotion intensity parameters and personality style parameters. It then uses a pre-trained DistilRoBERTa model to extract text emotion features, a WavLM model to extract speech content features, and an EDWavLM model to extract speech emotion features, respectively. These text emotion features and speech emotion features are then fused to obtain multimodal emotion features. Subsequently, the speech content features are input into the content encoder, and the multimodal emotion features are input into the emotion encoder. A cross-reconstruction strategy is used to decouple the features, making the content features and emotion features independent. Simultaneously, the emotion intensity parameters and personality style parameters are converted into explicit control features through an embedding layer, and concatenated with the decoupled content features and emotion features to obtain fused features. These fused features are then input into a Transformer-based emotion-driven decoder, which includes periodic positional encoding, an ALiBi biased multi-head self-attention mechanism, and a biased cross-modal attention mechanism. The decoder outputs a 52-dimensional facial hybrid shape coefficient sequence conforming to the ARKit standard, ultimately driving the digital human to present facial animation synchronized with speech and expressing emotion. This approach enhances the comprehensiveness of emotion understanding through multimodal fusion, resolves the mutual interference between semantics and emotion through feature decoupling, and achieves adjustability of emotion intensity and personality style through explicit control. The generated facial animations are natural and smooth, express emotions accurately, and have good platform portability, effectively improving the realism of digital human interaction. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating a digital human emotion simulation method based on multimodal data fusion.

[0070] Figure 2 This is a structural block diagram of a digital human emotion simulation device based on multimodal data fusion. Detailed Implementation

[0071] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0072] In this application, the execution entity is a control system. The system backend is deployed on an Ubuntu 20.04 server, and the frontend is an Unreal Engine client running Windows 10. User voice input is via microphone; the client calls ASR to convert it to text and sends it to the backend via WebSocket. The backend calls ChatGLM3-6B to generate reply text, and then calls EmotiVoice TTS to synthesize speech. Voice and text are simultaneously input into the emotion simulation module, generating 52-dimensional BlendShapes coefficients. Voice is streamed to the client via WebSocket, and coefficients are sent at 30fps via the OSC protocol. After receiving the data, the client drives the MetaHuman digital human to synchronously play voice and facial animations, meeting real-time interaction requirements.

[0073] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0074] S10: Acquire multimodal input data; use a pre-trained text sentiment feature extraction model to extract text sentiment features from the driving text; use a pre-trained speech content feature extraction model to extract speech content features from the speech signal; use a pre-trained speech sentiment feature extraction model to extract speech sentiment features from the speech signal; and perform feature-level fusion of text sentiment features and speech sentiment features to obtain multimodal sentiment features.

[0075] S20: Input the speech content features into the content encoder for encoding to obtain the decoupled content features; input the multimodal emotion features into the emotion encoder for encoding to obtain the decoupled emotion features. The content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other.

[0076] S30 converts the emotional intensity parameter and personality style parameter into continuous dense vectors through the embedding layer, and concatenates the two to obtain the explicit control feature; the decoupled content feature, emotional feature and explicit control feature are concatenated to obtain the fused feature.

[0077] S40 integrates the feature input to the Transformer-based emotion-driven decoder, which includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, thereby obtaining a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

[0078] The multimodal input data includes driving text, the corresponding speech signal, and user-defined emotional intensity parameters and personality style parameters.

[0079] Specifically, the system first acquires the driving text, corresponding speech, and user-defined emotional intensity and personality style parameters. It then extracts text emotion features, speech content features, and speech emotion features using a pre-trained model, fusing the text and speech emotion features into a multimodal emotion feature. Subsequently, a content encoder and an emotion encoder decouple the speech content features and multimodal emotion features, using a cross-reconstruction strategy to make the content features and emotion features independent. Simultaneously, the emotional intensity and personality style parameters are converted into explicit control features through an embedding layer, which are then concatenated with the decoupled content and emotion features to obtain the fused features. Finally, the fused features are input into a Transformer decoder containing biased multi-head self-attention and biased cross-modal attention mechanisms to generate a facial feature coefficient sequence, which is mapped to the digital human model, driving it to present emotionally rich expressions synchronized with the speech. Thus, through multimodal fusion, feature decoupling, and explicit control, the system effectively improves the realism, accuracy, and controllability of the digital human's emotional expression, enhancing the human-computer interaction experience.

[0080] In some embodiments, step S10 specifically includes the following steps: inputting speech content features into a content encoder, which consists of a linear layer and a 6-layer Transformer encoder, reducing the speech content features from 1024 dimensions to 512 dimensions, and outputting decoupled content features; inputting multimodal emotion features into an emotion encoder, which consists of two linear layers and a ReLU activation layer, reducing the multimodal emotion features from 832 dimensions to 256 dimensions, and outputting decoupled emotion features;

[0081] A cross-reconstruction strategy is employed to jointly train the content encoder and the emotion encoder. Two pairs of samples with different contents and emotions are selected from the training data, and their content features and emotion features are extracted respectively. The content features of the first sample are cross-combined with the emotion features of the second sample and input into the decoder to generate the first predicted facial animation. The content features of the second sample are cross-combined with the emotion features of the first sample and input into the decoder to generate the second predicted facial animation. With real facial animation as supervision, the cross-reconstruction loss function is minimized to force the content features and emotion features to be independent of each other in the feature space, making the content features and emotion features orthogonal or approximately orthogonal in the feature space.

[0082] In practice, the cross-reconstruction strategy is as follows: First and second samples are selected from the training dataset. The first sample contains first semantic content, a first emotion category, and its corresponding first real facial animation; the second sample contains second semantic content, a second emotion category, and its corresponding second real facial animation. The training dataset contains various combinations of samples with the same semantic content but different emotion categories, as well as samples with the same emotion category but different semantic content. The first and second samples are input into the content encoder and emotion encoder, respectively, to extract the content and emotion features of the first and second samples. The content features of the first sample and the emotion features of the second sample are concatenated and input into the decoder to generate the first predicted facial animation. The content features of the second sample and the emotion features of the first sample are concatenated and input into the decoder to generate the second predicted facial animation. The first real facial animation corresponding to the first semantic content and the second emotion category, and the second real facial animation corresponding to the second semantic content and the first emotion category, are obtained from the training dataset. The cross-reconstruction loss is calculated as the sum of the squared Euclidean distance between the first predicted facial animation and the first real facial animation, plus the squared Euclidean distance between the second predicted facial animation and the second real facial animation. The parameters of the content encoder and the emotion encoder are optimized through backpropagation.

[0083] This decouples content features from emotion features in the feature space, allowing any combination of content features and emotion features to decode facial animations that are consistent with the target semantics and emotions, thereby achieving orthogonal separation of semantic and emotional information.

[0084] The content encoder consists of one linear layer (1024→512) and a 6-layer Transformer encoder, with each hidden layer having a dimension of 512 and 8 multi-head attention heads. The emotion encoder consists of two linear layers (832→512, 512→256), with ReLU activations inserted in between. During training, two samples with different content and different emotions are randomly selected from the 3D-ETF dataset. Content features c1 and c2 and emotion features e1 and e2 are extracted respectively, and cross-combined into [c1;e2] and [c2;e1] input decoders. The B-values ​​are then calculated to match the actual animation. 12 B 21 The L1 loss is used as the cross-reconstruction loss.

[0085] Specifically, to decouple the speech features of the content encoder and the emotion encoder, a cross-reconstruction strategy is adopted for joint training: different content and emotion sample pairs are selected, and the content features of the first sample are cross-combined with the emotion features of the second sample to generate predicted animations, and vice versa. Real animations are used as supervision to minimize the cross-reconstruction loss. This strategy forces the model to separate semantic and emotional information, making the content features and emotion features orthogonal or approximately orthogonal in the feature space. Its core function is to eliminate the mutual interference between semantic and emotional information in speech, ensuring that content features focus on lip-driven features and emotion features are responsible for facial expression generation, laying the foundation for subsequent controllable facial animation.

[0086] In some embodiments, step S30 specifically includes the following steps: inputting the emotion intensity parameter into the first embedding layer, the emotion intensity parameter being a two-dimensional one-hot encoded vector, representing two discrete categories of high-intensity emotion and low-intensity emotion respectively, and mapping it to a 32-dimensional continuous dense vector through the first embedding layer to obtain the emotion intensity embedding feature; inputting the personality style parameter into the second embedding layer, the personality style parameter being a 24-dimensional one-hot encoded vector, representing 24 different personality style categories respectively, and mapping it to a 32-dimensional continuous dense vector through the second embedding layer to obtain the personality style embedding feature; concatenating the emotion intensity embedding feature and the personality style embedding feature into vectors to obtain a 64-dimensional explicit control feature, the explicit control feature continuously optimizing the embedding matrix parameters through backpropagation during training, so that semantically similar categories are closer in the embedding space;

[0087] The decoupled content features, emotion features, and explicit control features are concatenated. The content features are 512-dimensional, the emotion features are 256-dimensional, and the explicit control features are 64-dimensional. After concatenation, an 832-dimensional fusion feature is obtained. The fusion feature simultaneously includes fine-grained temporal content information of speech, global emotion information, and user-adjustable emotional intensity and personality style information.

[0088] Specifically, firstly, two-dimensional one-hot encoding of emotional intensity (high intensity / low intensity) is input into the first embedding layer, mapping it to a 32-dimensional emotional intensity embedding feature; secondly, 24-dimensional one-hot encoding of personality style is input into the second embedding layer, mapping it to a 32-dimensional personality style embedding feature; the two are concatenated to obtain a 64-dimensional explicit control feature. This feature optimizes the embedding matrix during training through backpropagation, making similar categories closer in the embedding space. Subsequently, the decoupled 512-dimensional content feature, 256-dimensional emotion feature, and 64-dimensional explicit control feature are concatenated to obtain an 832-dimensional fusion feature. This transforms discrete user control parameters into learnable continuous representations, enabling the fusion feature to simultaneously contain fine-grained temporal content of speech, global emotional information, and user-adjustable emotional intensity and personality style, providing the decoder with comprehensive multi-dimensional input and realizing personalized emotional expression in digital humans.

[0089] In some embodiments, step S40 specifically includes the following steps: constructing a Transformer-based emotion-driven decoder, which consists of multiple stacked decoding layers. Each decoding layer includes a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer uses residual connections and layer normalization. Periodic position encoding is added to the fused features to obtain encoded features carrying position information. The periodic position encoding is generated using sine and cosine functions, with its period parameter set to sixteen. By taking the position index modulo the period and calculating the trigonometric function value, the model can perceive the motion patterns between frames and reduce lip-sync drift in long sequences.

[0090] For the pos-th time step, the 2i-th dimension, and the 2i+1-th dimension, the position encoding values ​​are as follows:

[0091] ;

[0092] ;

[0093] Where pos is the time step index, i is the dimension index, d is the feature dimension (d=832 here), and p is the period parameter (p=16 here). This encoding calculates the trigonometric function value by taking the position index modulo the period.

[0094] Next, the system inputs the encoded features carrying location information into the biased multi-head self-attention sub-layer of the first decoding layer. In the biased multi-head self-attention sub-layer, the ALiBi mechanism is used to add a bias term that is linearly related to the distance between the query and the key to the scaled dot product attention score. The bias slope is set to the reciprocal of the head index plus one, so that different attention heads penalize distant positions at different rates, resulting in the self-attention output. The calculation formula is as follows:

[0095] ;

[0096] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. For the dimension of the key, This represents the distance between query position i and key position j, and m is a head-specific slope, set to the reciprocal of the head index plus one. This allows different attention heads to penalize distant positions at different rates, thereby enhancing the model's generalization ability to variable-length sequences.

[0097] Then, the system performs a residual concatenation between the self-attention output and the input of the biased multi-head self-attention sub-layer, followed by layer normalization, to obtain the first sub-layer output. The first sub-layer output is used as a query, and the emotion feature sequence is used as the key and value, inputting into the biased cross-modal multi-head attention sub-layer. Simultaneously, a memory mask matrix is ​​constructed, where the memory mask matrix takes a value of zero only at the emotion feature positions within the current time step and a preset time window, and a value of negative infinity at other positions. This memory mask matrix is ​​then introduced into the attention calculation to ensure that the attention mechanism only focuses on temporally aligned emotion features, resulting in the cross-modal attention output. The cross-modal attention output is then residually concatenated with the first sub-layer output and layer normalized, yielding the second sub-layer output. The second sub-layer output is input into the feedforward neural network sub-layer for nonlinear transformation, resulting in the feedforward output. Finally, the feedforward output is residually concatenated with the second sub-layer output and layer normalized, yielding the final output of the current decoding layer.

[0098] Memory mask matrix Its elements are defined as:

[0099] ;

[0100] By incorporating the memory mask matrix into the attention calculation, the biased cross-modal attention output is obtained, as shown in the following formula:

[0101] ;

[0102] Where i is the query time step index, j is the key time step index, and k is the preset window scaling factor, representing the number of emotion feature frames contained in the time window corresponding to each facial animation frame. This is the query from the current layer of the decoder (i.e., the output of the first sub-layer). These are the keys and values ​​for the emotional characteristics, respectively. The dimension is the key. This mechanism enables precise alignment between modalities.

[0103] Finally, the system uses the final output of the current decoding layer as the input of the next decoding layer, repeating the decoding steps until the last decoding layer outputs the final feature sequence. The feature sequence output by the last decoding layer is then input into a fully connected mapping layer, which reduces the feature dimension from 832 to 52 and constrains the output value to between zero and one using the Sigmoid activation function, thus obtaining the facial feature coefficient sequence corresponding to the time step.

[0104] Specifically, the decoder consists of multiple stacked decoding layers. Each layer contains a biased multi-head self-attention sublayer, a biased cross-modal multi-head attention sublayer, and a feedforward neural network sublayer. Each sublayer employs residual connections and layer normalization. First, a periodic positional encoding with a period of sixteen is added to the fused features. Trigonometric function values ​​are calculated by taking the modulus of the period using the position index, enabling the model to perceive inter-frame motion patterns and prevent lip-sync drift in long sequences. The encoded features are then input into the first decoding layer. The biased multi-head self-attention sublayer employs the ALiBi mechanism, adding a bias term linearly related to distance by head index, enhancing generalization ability for variable-length sequences. Subsequently, the self-attention output is used as a query, and the emotion feature sequence is used as the key and value input to the biased cross-modal multi-head attention sublayer. A memory mask matrix is ​​introduced to force each facial frame to focus only on emotion features within the temporal alignment window, achieving precise modal alignment. The cross-modal attention output is processed by residual connections and a feedforward network to obtain the current layer output and is then passed to the next layer. The output of the final decoding layer is reduced from 832 to 52 dimensions via a fully connected mapping layer, and then activated by a Sigmoid function to obtain a sequence of facial feature coefficients. This decoder effectively models long-term temporal dependencies through a biased attention mechanism and modal alignment design, ensuring that the generated facial animation is temporally coherent, emotionally accurate, and lip-synced, providing driving parameters for the emotional expression of digital humans.

[0105] In some embodiments, considering the specific implementation mechanism of the biased cross-modal multi-head attention sublayer, in order to achieve accurate temporal alignment between emotion features and facial motion features, the corresponding processing steps are as follows: The output of the first sublayer obtained after residual connection and layer normalization is used as the query input, where each row of the query input corresponds to a decoded feature at one time step; the decoupled emotion feature sequence is used as the key and value inputs, with the frame rate of the emotion feature sequence matching the frame rate of the facial animation, and each frame corresponding to an emotion feature vector; a memory mask matrix is ​​constructed, where the number of rows in the memory mask matrix equals the number of time steps in the query input, and the number of columns equals the number of time steps in the emotion feature sequence; for each element in the memory mask matrix, if the column index is greater than or equal to the preset window scaling factor multiplied by the row index, and less than the preset window scaling factor multiplied by the row index plus one, then the corresponding element takes the value of zero; otherwise, it takes the value of negative infinity; where the preset window scaling factor represents the number of emotion feature frames contained within the time window corresponding to each facial animation frame;

[0106] The query input is linearly projected onto multiple attention heads to obtain a query submatrix for each head. The key and value inputs are linearly projected onto the same number of attention heads to obtain a key submatrix and a value submatrix for each head. For each attention head, the product of the query submatrix and the transpose of the key matrix is ​​calculated, and then divided by the square root of the key matrix dimension to obtain a scaled dot product attention fractional submatrix. The memory mask matrix is ​​added to the scaled dot product attention fractional submatrix to obtain a biased attention fractional submatrix. The biased attention fractional submatrix is ​​normalized row-wise to obtain an attention weight submatrix. The attention weight submatrix is ​​multiplied by the value submatrix to obtain the attention output for each head. All attention outputs are concatenated and then fused through a linear projection layer to obtain a cross-modal attention output, which is used as the calculation result of this sublayer.

[0107] It should be understood that the memory mask matrix achieves precise alignment between emotional features and facial motion features by forcing each facial animation frame to focus only on the emotional features within its time-aligned window. This avoids emotional features at irrelevant time points interfering with the expression generation of the current frame, thereby improving the consistency of emotional expression and temporal coherence of facial animation.

[0108] Specifically, the output of the first sub-layer after residual connection and layer normalization is used as the query input, with each row corresponding to the decoded features of one time step. The decoupled emotion feature sequence is used as the key and value input, with the frame rate of the emotion feature sequence consistent with the frame rate of the facial animation. The core is the construction of a memory mask matrix, where the number of rows equals the number of query time steps and the number of columns equals the number of emotion feature time steps. For each element in the matrix, if the column index is within the window range determined by the preset window scaling factor multiplied by the row index, the value is zero; otherwise, the value is negative infinity. The window scaling factor represents the number of emotion feature frames contained within the time window corresponding to each facial animation frame. In the multi-head attention calculation, the query, key, and value are linearly projected onto multiple attention heads. The scaled dot product attention score is calculated for each head and added to the memory mask matrix, so that the attention score outside the window is set to negative infinity, resulting in a weight of zero after softmax, ensuring that each facial frame only focuses on the emotion features within the time-aligned window. Finally, the outputs of all heads are concatenated and fused through linear projection to obtain the cross-modal attention output.

[0109] By forcibly achieving precise time alignment between modalities through a memory mask matrix, the interference of emotional features from irrelevant time points with the generation of expressions in the current frame is effectively avoided, significantly improving the consistency of emotional expression and temporal coherence in facial animation.

[0110] In some embodiments, to apply periodic positional encoding to the fused features to endow the model with time-series awareness, the corresponding processing steps are as follows: The fused features are used as the input feature sequence, with a dimension of 832 and a time step count of T, where each frame corresponds to a fused feature vector; a periodic positional encoding matrix is ​​constructed, with the same dimension as the input feature sequence, consisting of T rows and 832 columns; for the positional encoding values ​​at the pos-th time step, the 2i-th dimension, and the (2i+1)-th dimension, the encoding value of the 2i-th dimension is equal to the reciprocal of an exponential function with the natural constant e as its base. The periodic function value of the denominator is obtained by multiplying the exponent 2i by 832 and then by the natural logarithm, and the numerator is obtained by taking the position index pos modulo the periodic parameter 16 as the sine function value of the independent variable. The encoded value of the (2i+1)th dimension is equal to the reciprocal of the exponential function with the natural constant e as the base, which is used as the periodic function value of the denominator. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm, and the numerator is obtained by taking the position index pos modulo the periodic parameter 16 as the cosine function value of the independent variable. The periodic position encoding matrix is ​​added element by element to the input feature sequence to obtain the encoded features carrying position information.

[0111] Specifically, the 832-dimensional fused feature sequence is used as input to construct a periodic position encoding matrix with the same dimensions. For each time step pos and each feature dimension pair... The encoded value is calculated by taking the sine and cosine values ​​of the position index pos modulo 16, respectively, and adjusting the frequency of different dimensions using an exponential function with the natural constant e as the base. The encoded matrix is ​​then added element-wise to the input features to obtain the encoded features carrying positional information. This compresses the infinite positional space into a finite period, enabling the model to perceive relative positions and motion patterns between frames, while avoiding the problem of excessively large or small positional encoded values ​​in long sequences. This effectively reduces lip-sync drift and improves the temporal coherence and generation stability of facial animation.

[0112] It should be understood that periodic positional coding, by compressing an infinite positional space into a finite period, enables the model to perceive the relative positional relationships between frames. Simultaneously, it avoids the problem of excessively large or small positional coding values ​​due to excessively long sequence lengths, thereby maintaining the stability of lip movements during the generation of long speech sequences and effectively reducing lip drift.

[0113] In some embodiments, a multimodal emotion-driven model is constructed and trained; the multimodal emotion-driven model includes a feature extraction module, a feature decoupling module, an explicit control embedding module, and an emotion-driven decoder; the feature extraction module includes a text emotion feature extraction model, a speech content feature extraction model, and a speech emotion feature extraction model; the feature decoupling module includes a content encoder and an emotion encoder; and the explicit control embedding module is used to generate explicit control features.

[0114] The training process includes the following steps: Constructing a training dataset containing multiple sample pairs, each pair including driving text, speech signal, corresponding real facial feature coefficient sequences, and emotion category labels; the training dataset contains various combinations of samples with the same semantic content but different emotion categories, and the same emotion category but different semantic content; initializing the feature extraction module: the text sentiment feature extraction model uses the DistilRoBERTa model pre-trained on the sentiment dataset, the speech content feature extraction model uses the WavLM model pre-trained on a large unsupervised speech dataset, and the speech emotion feature extraction model uses the EDWavLM model fine-tuned on the sentiment dataset; during training, the parameters of the feature extraction module are fixed, and it is used only as a feature extractor; initializing the feature decoupling module: the content encoder consists of a linear layer and a six-layer Transformer encoder, reducing the speech content features from 1024 dimensions to 512 dimensions; the emotion encoder consists of two linear layers and a ReLU activation layer, reducing the multimodal emotion features from 832 dimensions to 256 dimensions.

[0115] The explicit control embedding module is initialized, which includes a first embedding layer and a second embedding layer. The first embedding layer maps two-dimensional emotion intensity one-hot encoding to thirty-two-dimensional emotion intensity embedding features. The second embedding layer maps twenty-four-dimensional personality style one-hot encoding to thirty-two-dimensional personality style embedding features. The emotion-driven decoder is initialized, which consists of multiple stacked decoding layers and fully connected mapping layers. Each decoding layer includes a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer uses residual connections and layer normalization. The fully connected mapping layer maps the eight hundred and thirty-two-dimensional features output by the decoder to fifty-two-dimensional facial feature coefficients. A first sample and a second sample are randomly selected from the training dataset. The first sample contains a first semantic content, a first emotion category, and its corresponding first real facial feature coefficient sequence. The second sample contains a second semantic content, a second emotion category, and its corresponding second real facial feature coefficient sequence.

[0116] The driving text and speech signals of the first and second samples are input into the feature extraction module to extract the text emotion features, speech content features, and speech emotion features of the first sample, as well as the text emotion features, speech content features, and speech emotion features of the second sample. The text emotion features and speech emotion features of the first sample are fused to obtain the first multimodal emotion feature, and the text emotion features and speech emotion features of the second sample are fused to obtain the second multimodal emotion feature. The speech content features of the first sample are input into the content encoder to obtain the first content feature. The first multimodal emotion features of the first sample are input into the emotion encoder to obtain the first emotion feature. The speech content features of the second sample are input into the content encoder to obtain the second content feature. The second multimodal emotion features of the second sample are input into the emotion encoder to obtain the second emotion feature.

[0117] The first content feature and the second emotion feature are concatenated and input into the emotion-driven decoder to generate a first predicted facial feature coefficient sequence; the second content feature and the first emotion feature are concatenated and input into the emotion-driven decoder to generate a second predicted facial feature coefficient sequence; the first true facial feature coefficient sequence corresponding to the first semantic content and the second emotion category, and the second true facial feature coefficient sequence corresponding to the second semantic content and the first emotion category are obtained from the training dataset; the joint loss function value is calculated. The weighted sum of the cross-reconstruction loss, self-reconstruction loss, velocity loss, and classification loss is calculated as follows:

[0118] ;

[0119] in, These are preset coefficients.

[0120] Cross-reconstruction loss for: ;

[0121] in, and These are the first and second predicted facial feature coefficient sequences generated after cross-combination, respectively. and This corresponds to the actual sequence.

[0122] Self-reconfiguration loss for: ;

[0123] in, and These are the prediction sequences obtained by concatenating the first content feature with the first emotion feature, and the second content feature with the second emotion feature, respectively, and then inputting them into the decoder. and This corresponds to the actual sequence.

[0124] speed loss for: ;

[0125] in, and These are the predicted and actual facial feature coefficients for frame t, respectively. This loss constrains the rate of change between adjacent frames, improving animation smoothness.

[0126] Classification loss Cross-entropy loss: ;

[0127] Where N is the number of samples and C is the number of emotion categories. This is an indicator variable (1 if sample i belongs to category c, 0 otherwise). The loss is used to predict the probability that sample i belongs to category c for the model. This loss is used to enhance the emotion encoder's ability to distinguish emotion categories.

[0128] The cross-reconstruction loss is the sum of the squared Euclidean distance between the first predicted facial feature coefficient sequence and the first real facial feature coefficient sequence, plus the sum of the squared Euclidean distances between the second predicted facial feature coefficient sequence and the second real facial feature coefficient sequence; the self-reconstruction loss is the sum of the squared Euclidean distance between the predicted sequence obtained by concatenating the first content feature and the first emotion feature and inputting it into the emotion-driven decoder, and the first real facial feature coefficient sequence, plus the sum of the squared Euclidean distances between the predicted sequence obtained by concatenating the second content feature and the second emotion feature and inputting it into the emotion-driven decoder, and the second real facial mixed shape coefficient sequence; the velocity loss is the squared Euclidean distance between the difference vector between adjacent frames of the predicted sequence and the difference vector between adjacent frames of the real sequence; the first emotion feature and the second emotion feature are respectively input into the classifier to obtain the probability distribution of the predicted emotion category, and the sum of the cross-entropy losses with the real emotion category labels is calculated to obtain the classification loss; based on the joint loss function value, the parameters of the content encoder, emotion encoder, explicit control embedding module, and emotion-driven decoder are updated through the backpropagation algorithm;

[0129] Repeat the training steps until the joint loss function value converges or the preset number of training rounds is reached to obtain a trained multimodal emotion-driven model.

[0130] Specifically, text sentiment feature extraction uses the DistilRoBERTa-base model, fine-tuned on the RAVDESS sentiment dataset, outputting a 768-dimensional text feature vector. Speech content feature extraction uses the WavLM-Large model, outputting a 1024-dimensional feature sequence at a frame rate of 50Hz. Speech emotion feature extraction uses the EDWavLM model, fine-tuned on RAVDESS, also outputting a 1024-dimensional feature sequence. The text features and speech emotion features are concatenated to obtain multimodal emotion features with a dimension of 1792, which are then reduced to 832 using a linear layer.

[0131] The Transformer decoder consists of four layers, each containing a biased multi-head self-attention layer (8 heads), a biased cross-modal multi-head attention layer (8 heads), and a feedforward network. The ALiBi bias slope m is set to 1 / sqrt(head index + 1). The periodic position encoding period p = 16. Finally, a fully connected layer (832 → 52) outputs 52-dimensional BlendShapes coefficients. The Adam optimizer is used with a learning rate of 1e-4, a batch size of 8, and training for 100 epochs.

[0132] The implementation principle of the digital human emotion simulation method based on multimodal data fusion in this application is as follows: First, the driving text, corresponding speech signal, and user-defined emotion intensity parameters and personality style parameters are acquired. Pre-trained DistilRoBERTa models are used to extract text emotion features, WavLM models to extract speech content features, and EDWavLM models to extract speech emotion features, respectively. The text emotion features and speech emotion features are then fused to obtain multimodal emotion features. Subsequently, the speech content features are input into the content encoder, and the multimodal emotion features are input into the emotion encoder. A cross-reconstruction strategy is used to decouple the features, making the content features and emotion features independent. Simultaneously, the emotion intensity parameters and personality style parameters are converted into explicit control features through an embedding layer, and concatenated with the decoupled content features and emotion features to obtain fused features. The fused features are input into a Transformer-based emotion-driven decoder, which includes periodic positional encoding, an ALiBi biased multi-head self-attention mechanism, and a biased cross-modal attention mechanism. The decoder outputs a 52-dimensional facial hybrid shape coefficient sequence conforming to the ARKit standard, ultimately driving the digital human to present facial animation synchronized with speech and with emotional expression. This approach enhances the comprehensiveness of emotion understanding through multimodal fusion, resolves the mutual interference between semantics and emotion through feature decoupling, and achieves adjustability of emotion intensity and personality style through explicit control. The generated facial animations are natural and smooth, express emotions accurately, and have good platform portability, effectively improving the realism of digital human interaction.

[0133] Figure 1This is a flowchart illustrating a digital human emotion simulation method based on multimodal data fusion in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0134] Based on the same technical concept, referring to Figure 2 This application also provides a digital human emotion simulation device based on multimodal data fusion, which adopts the following technical solution: The device includes:

[0135] The feature extraction module is used to acquire multimodal input data, which includes driving text, the corresponding speech signal, and user-defined emotion intensity parameters and personality style parameters. It extracts text emotion features from the driving text using a pre-trained text emotion feature extraction model; extracts speech content features from the speech signal using a pre-trained speech content feature extraction model; extracts speech emotion features from the speech signal using a pre-trained speech emotion feature extraction model; and fuses the text emotion features and speech emotion features at the feature level to obtain multimodal emotion features.

[0136] The feature encoding module is used to input speech content features into the content encoder for encoding to obtain decoupled content features; and to input multimodal emotion features into the emotion encoder for encoding to obtain decoupled emotion features. The content encoder and emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other.

[0137] The feature fusion module is used to convert the emotion intensity parameter and personality style parameter into continuous dense vectors through the embedding layer, and then concatenate the two to obtain the explicit control feature; the decoupled content feature, emotion feature and explicit control feature are concatenated to obtain the fused feature;

[0138] The emotion-driven module is used to input fused features into the Transformer-based emotion-driven decoder. The emotion-driven decoder includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, thereby obtaining a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

[0139] In some embodiments, the feature encoding module is specifically used to input speech content features into the content encoder, which consists of a linear layer and a 6-layer Transformer encoder to reduce the speech content features from 1024 dimensions to 512 dimensions and output the decoupled content features.

[0140] The multimodal emotion features are input into the emotion encoder, which consists of two linear layers and a ReLU activation layer. The multimodal emotion features are reduced from 832 dimensions to 256 dimensions, and the decoupled emotion features are output.

[0141] A cross-reconstruction strategy is employed to jointly train the content encoder and the emotion encoder. Two pairs of samples with different contents and emotions are selected from the training data, and their content features and emotion features are extracted respectively. The content features of the first sample are cross-combined with the emotion features of the second sample and input into the decoder to generate the first predicted facial animation. The content features of the second sample are cross-combined with the emotion features of the first sample and input into the decoder to generate the second predicted facial animation. With real facial animation as supervision, the cross-reconstruction loss function is minimized to force the content features and emotion features to be independent of each other in the feature space, making the content features and emotion features orthogonal or approximately orthogonal in the feature space.

[0142] In some embodiments, the feature fusion module is specifically used to input the emotion intensity parameter into the first embedding layer. The emotion intensity parameter is a two-dimensional one-hot encoded vector, which represents two discrete categories: high-intensity emotion and low-intensity emotion. The first embedding layer maps it into a 32-dimensional continuous dense vector to obtain the emotion intensity embedded feature.

[0143] The personality style parameters are input into the second embedding layer. The personality style parameters are 24-dimensional one-hot encoded vectors, which represent 24 different personality style categories. The second embedding layer maps them into 32-dimensional continuous dense vectors to obtain the personality style embedding features.

[0144] The emotional intensity embedding feature and the personality style embedding feature are concatenated into vectors to obtain a 64-dimensional explicit control feature. During training, the explicit control feature continuously optimizes the embedding matrix parameters through backpropagation, so that semantically similar categories are closer together in the embedding space.

[0145] The decoupled content features, emotion features, and explicit control features are concatenated. The content features are 512-dimensional, the emotion features are 256-dimensional, and the explicit control features are 64-dimensional. After concatenation, an 832-dimensional fusion feature is obtained. The fusion feature simultaneously includes fine-grained temporal content information of speech, global emotion information, and user-adjustable emotional intensity and personality style information.

[0146] In some embodiments, the emotion-driven module is specifically used to construct a Transformer-based emotion-driven decoder. The emotion-driven decoder consists of multiple stacked decoding layers. Each decoding layer includes a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer employs residual connections and layer normalization.

[0147] Periodic positional coding is added to the fused features to obtain coded features carrying positional information. The periodic positional coding is generated using sine and cosine functions, with its period parameter set to sixteen. By taking the position index modulo the period and calculating the trigonometric function value, the model can perceive the motion pattern between frames and reduce lip-sync drift in long sequences.

[0148] The encoded features carrying location information are input into the biased multi-head self-attention sub-layer of the first decoding layer. In the biased multi-head self-attention sub-layer, the ALiBi mechanism is adopted to add a bias term that is linearly related to the distance between the query and the key to the scaled dot product attention score. The bias slope is set to the reciprocal of the head index plus one according to the attention head index, so that different attention heads penalize distant positions at different rates, and the self-attention output is obtained.

[0149] The self-attention output is residually connected to the input of the biased multi-head self-attention sub-layer and then layer normalized to obtain the first sub-layer output.

[0150] The output of the first sub-layer is used as the query, and the emotion feature sequence is used as the key and value, which are then input into the biased cross-modal multi-head attention sub-layer. At the same time, a memory mask matrix is ​​constructed, which takes a value of zero only at the emotion feature positions within the current time step and the preset time window, and a value of negative infinity at other positions. The memory mask matrix is ​​introduced into the attention calculation to ensure that the attention mechanism only focuses on temporally aligned emotion features, thus obtaining the cross-modal attention output.

[0151] The cross-modal attention output is residually connected to the output of the first sub-layer and then layer-normalized to obtain the output of the second sub-layer; the output of the second sub-layer is input into the feedforward neural network sub-layer for nonlinear transformation to obtain the feedforward output; the feedforward output is residually connected to the output of the second sub-layer and then layer-normalized to obtain the final output of the current decoding layer.

[0152] The final output of the current decoding layer is used as the input of the next decoding layer. The decoding steps are repeated until the last decoding layer outputs the final feature sequence.

[0153] The feature sequence output from the last decoding layer is input into the fully connected mapping layer. The fully connected mapping layer reduces the feature dimension from 832 to 52 and constrains the output value between zero and one through the Sigmoid activation function, thus obtaining the facial feature coefficient sequence corresponding to the time step.

[0154] In some embodiments, the emotion-driven module is specifically used to take the output of the first sub-layer obtained after residual connection and layer normalization as the query input, and each row of the query input corresponds to the decoding feature of a time step; and to take the decoupled emotion feature sequence as the key input and value input, the frame rate of the emotion feature sequence is consistent with the frame rate of the facial animation, and each frame corresponds to an emotion feature vector.

[0155] Construct a memory mask matrix, where the number of rows in the memory mask matrix equals the number of time steps of the query input, and the number of columns equals the number of time steps of the emotion feature sequence. For each element in the memory mask matrix, if the column index is greater than or equal to the preset window scaling factor multiplied by the row index, and less than the preset window scaling factor multiplied by the row index plus one, then the corresponding element takes the value of zero; otherwise, it takes the value of negative infinity. The preset window scaling factor represents the number of emotion feature frames contained in the time window corresponding to each facial animation frame.

[0156] The query input is linearly projected onto multiple attention heads to obtain the query submatrix for each head; the key input and value input are linearly projected onto the same number of attention heads to obtain the key submatrix and value submatrix for each head.

[0157] For each attention head, calculate the product of the query submatrix and the transpose of the key submatrix, and then divide by the square root of the dimension of the key matrix to obtain the scaled dot product attention fractional submatrix; add the memory mask matrix and the scaled dot product attention fractional submatrix to obtain the biased attention fractional submatrix.

[0158] The biased attention score submatrix is ​​normalized row by row to obtain the attention weight submatrix; the attention weight submatrix is ​​multiplied by the value submatrix to obtain the attention output of each head.

[0159] All attention outputs are concatenated and then fused through a linear projection layer to obtain cross-modal attention outputs, which are used as the computation results of this sub-layer.

[0160] In some embodiments, the emotion-driven module is specifically used to take the fused features as the input feature sequence, the input feature sequence having a dimension of 832, a time step of T, and each frame corresponding to a fused feature vector.

[0161] Construct a periodic positional encoding matrix with the same dimension as the input feature sequence, consisting of T rows and 832 columns. For the positional encoding values ​​at the pos-th time step, the 2i-th dimension, and the (2i+1)-th dimension, the encoding value of the 2i-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the sine function value of the independent variable. Similarly, the encoding value of the (2i+1)-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the cosine function value of the independent variable.

[0162] The periodic location coding matrix is ​​added element by element to the input feature sequence to obtain the coded features carrying location information.

[0163] In some embodiments, the emotion-driven module is specifically used to construct and train a multimodal emotion-driven model; the multimodal emotion-driven model includes a feature extraction module, a feature decoupling module, an explicit control embedding module, and an emotion-driven decoder; the feature extraction module includes a text emotion feature extraction model, a speech content feature extraction model, and a speech emotion feature extraction model; the feature decoupling module includes a content encoder and an emotion encoder; and the explicit control embedding module is used to generate explicit control features.

[0164] The training of a multimodal emotion-driven model includes the following steps:

[0165] Construct a training dataset containing multiple sample pairs. Each sample pair includes driving text, speech signal, corresponding real facial feature coefficient sequence, and emotion category label. The training dataset contains various combinations of samples with the same semantic content but different emotion categories, as well as samples with the same emotion category but different semantic content.

[0166] The feature extraction module is initialized. The text sentiment feature extraction model adopts the DistilRoBERTa model pre-trained on the sentiment dataset, the speech content feature extraction model adopts the WavLM model pre-trained on a large unsupervised speech dataset, and the speech emotion feature extraction model adopts the EDWavLM model fine-tuned on the sentiment dataset. During the training process, the parameters of the feature extraction module are fixed and it is only used as a feature extractor.

[0167] The initialization feature decoupling module consists of a content encoder composed of a linear layer and a six-layer Transformer encoder, which reduces the speech content features from 1,024 dimensions to 512 dimensions; the emotion encoder consists of two linear layers and a ReLU activation layer, which reduces the multimodal emotion features from 832 dimensions to 256 dimensions.

[0168] The explicit control embedding module is initialized. The explicit control embedding module includes a first embedding layer and a second embedding layer. The first embedding layer maps two-dimensional sentiment intensity one-hot encoding to thirty-two-dimensional sentiment intensity embedding features. The second embedding layer maps twenty-four-dimensional personality style one-hot encoding to thirty-two-dimensional personality style embedding features.

[0169] The emotion-driven decoder is initialized, which consists of multiple stacked decoding layers and fully connected mapping layers. Each decoding layer contains a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer uses residual connections and layer normalization. The fully connected mapping layer maps the 832-dimensional features output by the decoder to 52-dimensional facial feature coefficients.

[0170] First and second samples are randomly selected from the training dataset. The first sample contains first semantic content, first emotion category and its corresponding first real facial feature coefficient sequence, and the second sample contains second semantic content, second emotion category and its corresponding second real facial feature coefficient sequence.

[0171] The driving text and speech signals of the first and second samples are input into the feature extraction module to extract the text emotion features, speech content features, and speech emotion features of the first sample, as well as the text emotion features, speech content features, and speech emotion features of the second sample. The text emotion features and speech emotion features of the first sample are fused to obtain the first multimodal emotion feature, and the text emotion features and speech emotion features of the second sample are fused to obtain the second multimodal emotion feature.

[0172] The speech content features of the first sample are input into the content encoder to obtain the first content feature; the first multimodal emotion feature of the first sample is input into the emotion encoder to obtain the first emotion feature; the speech content features of the second sample are input into the content encoder to obtain the second content feature; the second multimodal emotion feature of the second sample is input into the emotion encoder to obtain the second emotion feature.

[0173] The first content feature and the second emotion feature are concatenated and input into the emotion-driven decoder to generate the first predicted facial feature coefficient sequence; the second content feature and the first emotion feature are concatenated and input into the emotion-driven decoder to generate the second predicted facial feature coefficient sequence.

[0174] Obtain the first true facial feature coefficient sequence corresponding to the first semantic content and the second emotion category from the training dataset, and the second true facial feature coefficient sequence corresponding to the second semantic content and the first emotion category.

[0175] Calculate the joint loss function value, which is a weighted sum of the cross-reconstruction loss, self-reconstruction loss, velocity loss, and classification loss;

[0176] The cross-reconstruction loss is the sum of the squared Euclidean distance between the first predicted facial feature coefficient sequence and the first true facial feature coefficient sequence, plus the squared Euclidean distance between the second predicted facial feature coefficient sequence and the second true facial feature coefficient sequence.

[0177] The self-reconstruction loss is the sum of the squared Euclidean distance between the predicted sequence obtained by concatenating the first content feature and the first emotion feature and inputting it into the emotion-driven decoder and the first real facial feature coefficient sequence, plus the squared Euclidean distance between the predicted sequence obtained by concatenating the second content feature and the second emotion feature and inputting it into the emotion-driven decoder and the second real facial mixed shape coefficient sequence.

[0178] The velocity loss is the squared Euclidean distance between the difference vectors between adjacent frames in the predicted sequence and the difference vectors between adjacent frames in the true sequence.

[0179] The first and second emotion features are input into the classifier respectively to obtain the probability distribution of the predicted emotion category. The sum of the cross-entropy loss between the predicted emotion category and the actual emotion category label is calculated to obtain the classification loss.

[0180] Based on the joint loss function value, the parameters of the content encoder, emotion encoder, explicit control embedding module, and emotion-driven decoder are updated using the backpropagation algorithm.

[0181] Repeat the training steps until the joint loss function value converges or the preset number of training rounds is reached to obtain a trained multimodal emotion-driven model.

[0182] This application also discloses a control device.

[0183] Specifically, the control device includes a memory and a processor, with the memory storing a computer program that can be loaded and executed by the processor to perform the aforementioned digital human emotion simulation method based on multimodal data fusion.

[0184] This application also discloses a computer-readable storage medium.

[0185] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as the digital human emotion simulation method based on multimodal data fusion described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A digital human emotion simulation method based on multimodal data fusion, characterized in that, include: The process involves acquiring multimodal input data, including driving text, corresponding speech signals, and user-defined emotion intensity and personality style parameters; extracting text emotion features from the driving text using a pre-trained text emotion feature extraction model; extracting speech content features from the speech signals using a pre-trained speech content feature extraction model; extracting speech emotion features from the speech signals using a pre-trained speech emotion feature extraction model; and fusing the text emotion features and the speech emotion features at the feature level to obtain multimodal emotion features. The speech content features are input into the content encoder for encoding to obtain decoupled content features; the multimodal emotion features are input into the emotion encoder for encoding to obtain decoupled emotion features. The content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other. The emotional intensity parameter and personality style parameter are respectively converted into continuous dense vectors through an embedding layer, and the two are concatenated to obtain the explicit control feature; the decoupled content feature, emotional feature and the explicit control feature are concatenated to obtain the fused feature; The fused features are input into a Transformer-based emotion-driven decoder, which includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, resulting in a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

2. The digital human emotion simulation method based on multimodal data fusion according to claim 1, characterized in that, The process involves inputting the speech content features into a content encoder for encoding to obtain decoupled content features; inputting the multimodal emotion features into an emotion encoder for encoding to obtain decoupled emotion features; and jointly training the content encoder and emotion encoder using a cross-reconstruction strategy to ensure that the content features and emotion features are independent of each other, including: The speech content features are input into the content encoder, which consists of a linear layer and a 6-layer Transformer encoder, reducing the speech content features from 1024 dimensions to 512 dimensions and outputting the decoupled content features. The multimodal emotion features are input into the emotion encoder, which consists of two linear layers and a ReLU activation layer, reducing the multimodal emotion features from 832 dimensions to 256 dimensions and outputting the decoupled emotion features. A cross-reconstruction strategy is employed to jointly train the content encoder and the emotion encoder. Two pairs of samples with different contents and emotions are selected from the training data, and their content features and emotion features are extracted respectively. The content features of the first sample are cross-combined with the emotion features of the second sample and input into the decoder to generate the first predicted facial animation. The content features of the second sample are cross-combined with the emotion features of the first sample and input into the decoder to generate the second predicted facial animation. With real facial animation as supervision, the cross-reconstruction loss function is minimized to force the content features and emotion features to be independent of each other in the feature space, making the content features and emotion features orthogonal or approximately orthogonal in the feature space.

3. The digital human emotion simulation method based on multimodal data fusion according to claim 2, characterized in that, The emotional intensity parameter and personality style parameter are respectively converted into continuous dense vectors through an embedding layer, and the two are concatenated to obtain explicit control features; The decoupled content features, emotion features, and explicit control features are concatenated to obtain fused features, including: The emotion intensity parameter is input into the first embedding layer. The emotion intensity parameter is a two-dimensional one-hot encoded vector, which represents two discrete categories: high-intensity emotion and low-intensity emotion. The first embedding layer maps it into a 32-dimensional continuous dense vector to obtain the emotion intensity embedding feature. The personality style parameters are input into the second embedding layer. The personality style parameters are 24-dimensional one-hot encoded vectors, which represent 24 different personality style categories. The second embedding layer maps them into 32-dimensional continuous dense vectors to obtain the personality style embedding features. The emotional intensity embedding feature and the personality style embedding feature are concatenated into vectors to obtain the 64-dimensional explicit control feature. During training, the explicit control feature continuously optimizes the embedding matrix parameters through backpropagation, so that semantically similar categories are closer in the embedding space. The decoupled content features, emotion features, and explicit control features are concatenated. The content features are 512-dimensional, the emotion features are 256-dimensional, and the explicit control features are 64-dimensional. The concatenated features are 832-dimensional. The concatenated features include fine-grained temporal content information of speech, global emotion information, and user-adjustable emotional intensity and personality style information.

4. The digital human emotion simulation method based on multimodal data fusion according to claim 3, characterized in that, The fused features are input into a Transformer-based emotion-driven decoder. This decoder includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long-term temporal dependencies and achieve modal alignment, resulting in a sequence of facial feature coefficients corresponding to each time step, including: A Transformer-based emotion-driven decoder is constructed, which consists of multiple stacked decoding layers. Each decoding layer contains a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer adopts residual connections and layer normalization. Periodic position coding is added to the fused features to obtain coded features carrying position information. The periodic position coding is generated using sine and cosine functions, with its period parameter set to sixteen. By taking the position index modulo the period and calculating the trigonometric function value, the model can perceive the motion pattern between frames and reduce lip-sync drift in long sequences. The encoded features carrying location information are input into the biased multi-head self-attention sub-layer of the first decoding layer. In the biased multi-head self-attention sub-layer, the ALiBi mechanism is adopted to add a bias term that is linearly related to the distance between the query and the key to the scaled dot product attention score. The bias slope is set to the reciprocal of the head index plus one according to the attention head index, so that different attention heads punish distant positions at different rates to obtain self-attention output. The self-attention output is residually connected to the input of the biased multi-head self-attention sub-layer and then layer normalized to obtain the first sub-layer output. The output of the first sub-layer is used as a query, and the emotional feature sequence is used as the key and value, which are then input into the biased cross-modal multi-head attention sub-layer. At the same time, a memory mask matrix is ​​constructed. The memory mask matrix takes a value of zero only at the emotional feature positions within the current time step and a preset time window, and a value of negative infinity at other positions. The memory mask matrix is ​​introduced into the attention calculation to ensure that the attention mechanism only focuses on temporally aligned emotional features, thus obtaining the cross-modal attention output. The cross-modal attention output is residually connected to the first sub-layer output and then layer-normalized to obtain the second sub-layer output; the second sub-layer output is input into the feedforward neural network sub-layer for nonlinear transformation to obtain the feedforward output; the feedforward output is residually connected to the second sub-layer output and then layer-normalized to obtain the final output of the current decoding layer. The final output of the current decoding layer is used as the input of the next decoding layer. The decoding steps are repeated until the last decoding layer outputs the final feature sequence. The feature sequence output from the last decoding layer is input into the fully connected mapping layer. The fully connected mapping layer reduces the feature dimension from 832 to 52 and constrains the output value between zero and one through the Sigmoid activation function to obtain the facial feature coefficient sequence corresponding to the time step.

5. The digital human emotion simulation method based on multimodal data fusion according to claim 4, characterized in that, The output of the first sub-layer is used as a query, and the emotional feature sequence is used as a key and value, which are then input into the biased cross-modal multi-head attention sub-layer. At the same time, a memory mask matrix is ​​constructed, which takes a value of zero only at the emotional feature positions within the current time step and a preset time window, and a value of negative infinity at the other positions. By incorporating a memory mask matrix into attention computation, the attention mechanism is ensured to focus only on temporally aligned emotional features, resulting in cross-modal attention outputs, including: The output of the first sub-layer obtained after residual connection and layer normalization is used as the query input. Each row of the query input corresponds to the decoding feature of a time step. The decoupled emotion feature sequence is used as the key input and value input. The frame rate of the emotion feature sequence is consistent with the frame rate of the facial animation. Each frame corresponds to an emotion feature vector. Construct a memory mask matrix, where the number of rows in the memory mask matrix equals the number of time steps of the query input, and the number of columns equals the number of time steps of the emotion feature sequence. For each element in the memory mask matrix, if the column index is greater than or equal to the preset window scaling factor multiplied by the row index, and less than the preset window scaling factor multiplied by the row index plus one, then the corresponding element takes the value of zero; otherwise, it takes the value of negative infinity. The preset window scaling factor represents the number of emotion feature frames contained in the time window corresponding to each facial animation frame. The query input is linearly projected onto multiple attention heads to obtain the query submatrix for each head; the key input and value input are linearly projected onto the same number of attention heads to obtain the key submatrix and value submatrix for each head. For each attention head, calculate the product of the query submatrix and the transpose of the key submatrix, and then divide by the square root of the dimension of the key matrix to obtain the scaled dot product attention fractional submatrix; add the memory mask matrix to the scaled dot product attention fractional submatrix to obtain the biased attention fractional submatrix. The biased attention score submatrix is ​​normalized row by row to obtain the attention weight submatrix; the attention weight submatrix is ​​multiplied by the value submatrix to obtain the attention output of each head; All the attention outputs are concatenated and then fused through a linear projection layer to obtain the cross-modal attention output, which serves as the calculation result of this sub-layer.

6. The digital human emotion simulation method based on multimodal data fusion according to claim 5, characterized in that, The step of adding periodic positional encoding to the fused features to obtain encoded features carrying positional information includes: The fused features are used as the input feature sequence, which has a dimension of 832 and a time step of T. Each frame corresponds to a fused feature vector. Construct a periodic positional encoding matrix with the same dimension as the input feature sequence, consisting of T rows and 832 columns. For the positional encoding values ​​at the pos-th time step, the 2i-th dimension, and the (2i+1)-th dimension, the encoding value of the 2i-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the sine function value of the independent variable. Similarly, the encoding value of the (2i+1)-th dimension is equal to the periodic function value in the denominator of an exponential function with the natural constant e as the base. The exponent in the denominator is 2i divided by 832 and then multiplied by the natural logarithm. The numerator is the value of the position index pos modulo the periodic parameter 16, which is used as the cosine function value of the independent variable. The periodic position encoding matrix is ​​added element by element to the input feature sequence to obtain the encoded features carrying position information.

7. A digital human emotion simulation method based on multimodal data fusion according to claim 6, characterized in that, The method further includes: Construct and train a multimodal emotion-driven model; the multimodal emotion-driven model includes a feature extraction module, a feature decoupling module, an explicit control embedding module, and an emotion-driven decoder; the feature extraction module includes: a text emotion feature extraction model, a speech content feature extraction model, and a speech emotion feature extraction model; the feature decoupling module includes: a content encoder and an emotion encoder; the explicit control embedding module is used to generate explicit control features; Training a multimodal emotion-driven model includes the following steps: Construct a training dataset containing multiple sample pairs. Each sample pair includes driving text, speech signal, corresponding real facial feature coefficient sequence, and emotion category label. The training dataset contains various combinations of samples with the same semantic content but different emotion categories, as well as samples with the same emotion category but different semantic content. The feature extraction module is initialized. The text sentiment feature extraction model adopts the DistilRoBERTa model pre-trained on the sentiment dataset, the speech content feature extraction model adopts the WavLM model pre-trained on a large unsupervised speech dataset, and the speech emotion feature extraction model adopts the EDWavLM model fine-tuned on the sentiment dataset. During the training process, the parameters of the feature extraction module are fixed and it is only used as a feature extractor. The initialization feature decoupling module consists of a content encoder composed of a linear layer and a six-layer Transformer encoder, which reduces the speech content features from 1,024 dimensions to 512 dimensions; the emotion encoder consists of two linear layers and a ReLU activation layer, which reduces the multimodal emotion features from 832 dimensions to 256 dimensions. The explicit control embedding module is initialized, which includes a first embedding layer and a second embedding layer. The first embedding layer maps two-dimensional emotion intensity one-hot encoding to thirty-two-dimensional emotion intensity embedding features. The second embedding layer maps twenty-four-dimensional personality style one-hot encoding to thirty-two-dimensional personality style embedding features. The emotion-driven decoder is initialized, which consists of multiple stacked decoding layers and fully connected mapping layers. Each decoding layer contains a biased multi-head self-attention sub-layer, a biased cross-modal multi-head attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer uses residual connections and layer normalization. The fully connected mapping layer maps the 832-dimensional features output by the decoder to 52-dimensional facial feature coefficients. First and second samples are randomly selected from the training dataset. The first sample contains first semantic content, first emotion category and its corresponding first real facial feature coefficient sequence, and the second sample contains second semantic content, second emotion category and its corresponding second real facial feature coefficient sequence. The driving text and speech signals of the first and second samples are input into the feature extraction module to extract the text emotion features, speech content features, and speech emotion features of the first sample, as well as the text emotion features, speech content features, and speech emotion features of the second sample. The text emotion features and speech emotion features of the first sample are fused to obtain the first multimodal emotion feature, and the text emotion features and speech emotion features of the second sample are fused to obtain the second multimodal emotion feature. The speech content features of the first sample are input into the content encoder to obtain the first content feature; the first multimodal emotion feature of the first sample is input into the emotion encoder to obtain the first emotion feature; the speech content features of the second sample are input into the content encoder to obtain the second content feature; the second multimodal emotion feature of the second sample is input into the emotion encoder to obtain the second emotion feature. The first content feature and the second emotion feature are concatenated and input into the emotion-driven decoder to generate the first predicted facial feature coefficient sequence; the second content feature and the first emotion feature are concatenated and input into the emotion-driven decoder to generate the second predicted facial feature coefficient sequence. Obtain the first true facial feature coefficient sequence corresponding to the first semantic content and the second emotion category from the training dataset, and the second true facial feature coefficient sequence corresponding to the second semantic content and the first emotion category. Calculate the joint loss function value, which is a weighted sum of the cross-reconstruction loss, self-reconstruction loss, velocity loss, and classification loss; The cross-reconstruction loss is the sum of the squared Euclidean distance between the first predicted facial feature coefficient sequence and the first true facial feature coefficient sequence, plus the squared Euclidean distance between the second predicted facial feature coefficient sequence and the second true facial feature coefficient sequence. The self-reconstruction loss is the sum of the squared Euclidean distance between the predicted sequence obtained by concatenating the first content feature and the first emotion feature and inputting it into the emotion-driven decoder and the first real facial feature coefficient sequence, plus the squared Euclidean distance between the predicted sequence obtained by concatenating the second content feature and the second emotion feature and inputting it into the emotion-driven decoder and the second real facial mixed shape coefficient sequence. The velocity loss is the squared Euclidean distance between the difference vectors between adjacent frames in the predicted sequence and the difference vectors between adjacent frames in the true sequence. The first and second emotion features are input into the classifier respectively to obtain the probability distribution of the predicted emotion category. The sum of the cross-entropy loss between the predicted emotion category and the actual emotion category label is calculated to obtain the classification loss. Based on the joint loss function value, the parameters of the content encoder, emotion encoder, explicit control embedding module, and emotion-driven decoder are updated using the backpropagation algorithm. Repeat the training steps until the joint loss function value converges or the preset number of training rounds is reached to obtain a trained multimodal emotion-driven model.

8. A digital human emotion simulation device based on multimodal data fusion, characterized in that, The device includes: The feature extraction module is used to acquire multimodal input data, which includes driving text, speech signals corresponding to the driving text, and user-defined emotion intensity parameters and personality style parameters. It extracts text emotion features from the driving text using a pre-trained text emotion feature extraction model; extracts speech content features from the speech signals using a pre-trained speech content feature extraction model; extracts speech emotion features from the speech signals using a pre-trained speech emotion feature extraction model; and performs feature-level fusion of the text emotion features and the speech emotion features to obtain multimodal emotion features. The feature encoding module is used to input the speech content features into the content encoder for encoding to obtain decoupled content features; and to input the multimodal emotion features into the emotion encoder for encoding to obtain decoupled emotion features. The content encoder and the emotion encoder are jointly trained through a cross-reconstruction strategy to make the content features and emotion features independent of each other. The feature fusion module is used to convert the emotion intensity parameter and the personality style parameter into continuous dense vectors through the embedding layer, and then concatenate the two to obtain the explicit control feature; the decoupled content feature, emotion feature and the explicit control feature are concatenated to obtain the fused feature; The emotion-driven module is used to input the fused features into a Transformer-based emotion-driven decoder. The emotion-driven decoder includes a biased multi-head self-attention mechanism and a biased cross-modal attention mechanism to model long temporal dependencies and achieve modal alignment, thereby obtaining a facial feature coefficient sequence corresponding to the time step. The facial feature coefficient sequence is then mapped to the facial control unit of the target digital human model to drive the digital human to simulate expressions that are synchronized with speech and have emotional expression.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images