Digital human interaction system and method based on multi-modal emotion recognition

Abstract

The embodiment of the invention provides a digital human interaction system and method based on multi-modal emotion recognition, and belongs to the technical field of digital human interaction. The system comprises a multi-modal sensing module used for collecting multi-modal data and preprocessing the multi-modal data to generate a standardized data stream; the cross-modal fusion and emotion recognition module is used for carrying out interactive modeling on the multi-modal features and outputting a current emotion label and emotion intensity; the reaction planning module is used for generating a composite reaction strategy; and the digital human rendering module is used for mapping the composite reaction strategy into control signals corresponding to the voice, the facial expression and the action respectively, and driving a digital human to execute corresponding voice output, facial expression change and limb action through the control signals so as to realize interaction. According to the method, multi-modal data are deeply fused through the cross-modal graph neural network and comparative learning, the weight is dynamically adjusted in combination with the modal confidence, and the emotion recognition accuracy and robustness are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital human interaction, in particular to a digital human interaction system and method based on multi-modal emotion recognition. BACKGROUND

[0002] In recent years, digital humans have gradually moved out of the entertainment field and entered high-demand scenarios such as medical treatment, government affairs, and education. However, the existing digital human interaction generally has the problem of "lack of emotion", which makes it difficult for the interaction performance to gain the trust of users. For example, in psychological counseling, if the digital human only answers with mechanical voice and cannot capture the emotions such as anxiety, depression, or excitement of the user, the service effect will be greatly reduced. Traditional emotion recognition methods often rely on a single modality (such as speech emotion recognition or text emotion analysis), lack of integration of multi-source emotional clues, and are difficult to accurately reflect complex emotional states. On the other hand, most of the current multi-modal methods use simple feature splicing or early fusion, which cannot fully exploit the temporal relationship and complementarity between different modalities, resulting in limited recognition performance. At the same time, the reaction generation of digital humans is mostly based on fixed rules or preset templates, which lacks dynamic adaptation to user emotions and is difficult to achieve "different for different people" in the interaction process. Therefore, there is an urgent need for a new system that can deeply integrate multi-modal information, dynamically model emotional state transitions, and drive digital humans to generate emotion-adaptive reactions. The solution proposed in the present application combines graph neural networks, cross-modal attention mechanisms, and emotion memory networks, not only improving the accuracy and robustness of emotion recognition, but also enabling digital humans to have emotion-driven interaction capabilities, significantly enhancing user experience. SUMMARY

[0003] The purpose of the present application is to provide a digital human interaction system and method based on multi-modal emotion recognition, which is used to drive digital humans to generate natural interaction with empathy ability through multi-modal emotion recognition and dynamic reaction planning, and is applied to scenarios such as psychological counseling and intelligent customer service.

[0004] In order to achieve the above-mentioned purpose, the present application provides a digital human interaction system based on multi-modal emotion recognition, comprising: a multi-modal perception module for collecting multi-modal data and preprocessing the multi-modal data to generate standardized data streams; a cross-modal fusion and emotion recognition module for receiving the standardized data streams and modeling multi-modal features, outputting current emotion labels and emotion intensity; a reaction planning module for generating a composite reaction strategy according to the current emotion labels, the emotion intensity and the historical emotional trajectory; a digital human rendering module for mapping the composite reaction strategy into control signals corresponding to voice, facial expression and action respectively, and driving the digital human to execute corresponding voice output, facial expression change and body movement to realize interaction through the control signals.

[0005] Optionally, the preprocessing of the multi-modal data to generate a standardized data stream comprises: performing denoising processing on the voice signal and extracting mel-frequency spectrum and prosody features; performing semantic vectorization processing on the text based on a pre-trained language representation model; extracting facial micro-expression and facial dynamic features from the visual signal through a three-dimensional convolution network; extracting limb action features from the posture signal based on a graph convolution network; extracting physiological features from the physiological signal through one-dimensional convolution and time series modeling; aligning and standardizing the features of each modality in the time dimension to generate a standardized data stream; aligning and standardizing the features of each modality in the time dimension to generate a standardized data stream.

[0006] Optionally, the cross-modal fusion and emotion recognition module comprises: a cross-modal graph neural network unit, a cross-modal contrast learning unit, a time sequence attention network unit, and a sequence smoothing unit; the cross-modal graph neural network unit receives the features of each modality output by the multi-modal perception module, and maps the features of each modality into independent sub-nodes in a graph structure, respectively, models the association relationship between the independent sub-nodes by performing a preset graph convolution operation and calling a configured learnable edge weight matrix; the cross-modal contrast learning unit constructs positive sample pairs and negative sample pairs based on the features of each modality in the model training stage, and uses a preset contrast loss function to perform a maximization constraint on the similarity of the modality features corresponding to the positive sample pairs, and simultaneously performs a minimization constraint on the similarity of the modality features corresponding to the negative sample pairs; the time sequence attention network unit receives the multi-modal fusion feature sequence output by the cross-modal graph neural network unit, and performs self-attention calculation on the multi-modal fusion feature sequence, and assigns different time dimension attention weights to the multi-modal fusion features at different time steps in the feature sequence through the self-attention mechanism; the sequence smoothing unit receives the emotion recognition label sequence output by the time sequence attention network unit, and performs correlation constraint on the emotion recognition label sequences at adjacent time steps through a preset sequence smoothing model, and outputs the current emotion label and emotion intensity.

[0007] Optionally, the cross-modal graph neural network unit comprises: a modality confidence evaluation sub-unit that calculates a confidence score of each modality based on a short-term consistency indicator, a confidence output by a pre-trained encoder, and / or a signal-to-noise ratio; and a dynamic edge weight updating sub-unit that performs adaptive regularization on the learnable edge weight matrix to weaken the weight value of the edge corresponding to the modality feature in the graph structure in response to the confidence score being lower than a predetermined threshold, or in response to a condition that the noise of the modality feature exceeds a predetermined noise threshold being detected, or a condition that the modality feature is missing being detected.

[0008] Optionally, the adaptive regularization on the learnable edge weight matrix comprises: obtaining an actual edge weight through double-scale weight maintenance.

[0009] Optionally, the generating the composite response strategy according to the current emotional label, the emotional intensity and the historical emotional trajectory comprises: modeling the historical emotional trajectory of the user by using an emotional memory network to predict a transfer trend of the emotional state of the user; generating a plurality of candidate composite response strategies by using a candidate strategy generator based on the predicted transfer trend of the emotional state of the user and the current emotional label by using a sequence-to-sequence neural network with an attention mechanism; and obtaining the composite response strategy by using a strategy planner to evaluate and select the candidate composite response strategies by a reinforcement learning method based on a value function.

[0010] Optionally, the strategy planner takes an interaction effect index as a reward signal, and the interaction effect index comprises a user emotional state improvement degree and a user participation degree improvement value.

[0011] Optionally, the composite response strategy comprises an emotional response intensity parameter, and the emotional response intensity parameter is in a positive correlation mapping relationship with the emotional intensity output by the cross-modal fusion and emotional recognition module.

[0012] Optionally, the digital human rendering module comprises: a speech synthesis unit configured to decode the composite response strategy by using a sequence-to-sequence neural network based on a Tacotron architecture to generate a speech control signal; a facial expression generation unit configured to generate a facial expression control signal based on a deformation parameterization model of a BlendShape basic expression component in combination with an attention weight associated with the current emotional label; a motion generation unit configured to construct a graph structure with human skeleton joints as nodes, and to predict a joint position trajectory of a next time point under the constraint of the composite response strategy by using a graph convolution network to generate a motion control signal; and a timing synchronization unit configured to register the speech control signal, the facial expression control signal and the motion control signal by using a dynamic time warping algorithm, and to compensate for unaligned parts by using a time stamp interpolation method.

[0013] In another aspect, the present application provides a digital human interaction method based on multi-modal emotional recognition, comprising: collecting multi-modal data, and preprocessing the multi-modal data to generate standardized data streams; receiving the standardized data streams, and modeling multi-modal features to output a current emotional label and an emotional intensity; generating a composite response strategy according to the current emotional label, the emotional intensity and a historical emotional trajectory; mapping the composite response strategy into control signals corresponding to speech, facial expression and motion respectively, and driving a digital human to perform corresponding speech output, facial expression change and limb motion to realize interaction by using the control signals.

[0014] The above technical solution deeply integrates multimodal data through cross-modal graph neural networks and contrastive learning, and dynamically adjusts weights based on modality confidence to improve the accuracy and robustness of emotion recognition. By relying on emotional memory networks and reinforcement learning to generate dynamic composite response strategies, it avoids the limitations of fixed templates, realizes personalized empathetic interaction, and significantly improves user trust and interaction experience.

[0015] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a diagram of a digital human interaction system.

[0017] Figure 2 This is a flowchart of digital human interaction based on multimodal emotion recognition.

[0018] Figure 3 This is a flowchart of multimodal data preprocessing. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1 -Appendix Figure 3 The specific implementation methods of the embodiments of the present invention will be described in detail below. It should be understood that the specific implementation methods described herein are only for illustrating and explaining the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0020] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0021] In the process of realizing this invention, the inventors of this application discovered that the prior art has the following defects: digital human lacks emotion, has superficial multimodal fusion, has fixed responses, and has poor user trust and experience.

[0022] Example 1 Reference Figures 1-3This is the first embodiment of the present invention, which provides a digital human interaction system based on multimodal emotion recognition. The digital human interaction system may include: a multimodal perception module for collecting multimodal data and preprocessing the multimodal data to generate a standardized data stream; a cross-modal fusion and emotion recognition module for receiving the standardized data stream, performing interactive modeling of multimodal features, and outputting the current emotion tag and emotion intensity; a response planning module for generating a composite response strategy based on the current emotion tag, emotion intensity, and historical emotion trajectory; and a digital human rendering module for mapping the composite response strategy into control signals corresponding to speech, facial expressions, and actions, respectively, and driving the digital human to perform corresponding speech output, facial expression changes, and body movements through the control signals to achieve interaction.

[0023] In this embodiment of the invention, the digital human interaction system first collects real-time user interaction data through a multimodal sensing device. (See reference...) Figure 3 Preprocessing multimodal data to generate a standardized data stream can include: S110: Perform denoising processing on the speech signal and extract Mel spectrum and prosodic features.

[0024] The process involves capturing raw audio through a high-sensitivity microphone, removing silence using endpoint detection, and enhancing speech with spectral subtraction and Wiener filtering to remove noise. Finally, a convolutional neural network (CNN) is used to extract Mel spectrogram features and analyze prosodic information such as fundamental frequency and energy.

[0025] S120: Perform semantic vectorization processing on the text based on a pre-trained language representation model.

[0026] The text data originates from conversations with the digital human, transcribed and timestamped in real time by a natural language processing engine. Deep semantic encoding is performed using pre-trained language models such as RoBERTa to accurately capture the emotional polarity and potential emotional cues within the sentences.

[0027] S130: Extract facial micro-expressions and facial dynamic features from visual signals using a 3D convolutional network.

[0028] The visual signals are first captured by a high-definition camera, then facial regions are obtained through facial key point detection, and dynamic micro-expressions and gaze changes are captured by optical flow tracking. The visual modality is extracted using a 3D convolutional network to extract the aforementioned facial micro-expression and motion features.

[0029] S140: Extract limb movement features from posture signals using a graph convolutional network.

[0030] The posture signal is obtained by extracting human skeleton point data (including head tilt, gestures and body posture) through the posture recognition module and related acquisition devices (such as cameras that capture body movements). The limb movement patterns are further extracted through human skeleton point detection and graph convolutional network (GCN) to form posture features that can be used for emotion recognition.

[0031] S150: Physiological features are extracted from physiological signals through one-dimensional convolution and temporal modeling.

[0032] Among them, physiological signals are collected by wearable devices such as smart bracelets, which can obtain physiological data such as heart rate and skin conductance response, providing physiological-level emotional cues for subsequent emotion recognition.

[0033] S160: Align and standardize the above modal features in the time dimension to generate a standardized data stream.

[0034] In this process, all modal data are synchronized across multiple streams through a unified timestamp management mechanism. An interpolation alignment algorithm is used to address frame rate inconsistencies, and standardization and dimensionality reduction are employed at the feature layer to ensure input consistency, resulting in a standardized data stream. This provides rich and high-quality foundational information for emotion recognition.

[0035] In this embodiment of the invention, the modality fusion and emotion recognition module includes: a cross-modal graph neural network unit, a cross-modal contrastive learning unit, a temporal attention network unit, and a sequence smoothing unit.

[0036] In this embodiment of the invention, the cross-modal graph neural network unit receives the modal features output by the multimodal perception module and maps each modal feature to an independent child node in the graph structure. By performing a preset graph convolution operation and calling the configured learnable edge weight matrix, the relationship between each independent child node is modeled.

[0037] In a preferred embodiment of the present invention, the cross-modal graph neural network unit includes a modality confidence evaluation subunit and a dynamic edge weight update subunit. The modality confidence evaluation subunit calculates the confidence score (which may use adaptively learned weights) for each modality based on a short-time consistency index (which may employ a 3-second window), the confidence score output by the pre-trained encoder, and / or the signal-to-noise ratio. The dynamic edge weight update subunit performs adaptive regularization on the learnable edge weight matrix in response to conditions such as a confidence score falling below a predetermined threshold, noise exceeding a predetermined noise threshold for the modality feature, or the detection of a missing modality feature. This reduces the weight value of the corresponding edge in the graph structure for that modality feature.

[0038] In a preferred embodiment of the present invention, to achieve fine complementarity and robust fusion at the modality level, the cross-modal graph neural network unit constructs each modality as a subgraph node in the graph structure. That is, modalities such as speech, text, vision, posture, and physiological signals each correspond to an independent subgraph node. Each subgraph node not only carries the temporal feature vector of that modality, but also retains the local structural information within that modality (e.g., multi-region features of the visual modality, the set of key points of the posture modality, multi-channel signals of the physiological modality, etc.), so as to express both the semantics within the modality and the cross-modal interface information at the node level. The association between subgraph nodes is represented by a learnable edge weight matrix, which is dynamically updated according to an adaptive mechanism during training and online inference to achieve complementary behavior of automatically increasing the weight of complementary modalities (e.g., vision or posture) when a certain modality (e.g., speech) degenerates or is missing.

[0039] In a preferred embodiment of the present invention, the adaptive mechanism for dynamic updating includes: for any two subgraph nodes i and j, first calculating the compatibility score using the lightweight compatibility mapping, the formula for calculating the compatibility score is as follows:

[0040] in, This represents the compatibility score between node i and node j at time t. A non-linear mapping function representing compatibility. and Both represent learnable linear projection matrices (weight matrices). and Let i and j represent the embedding representations (vectors) of nodes i and j at time t, respectively. Represents element-wise product. This indicates the bias term.

[0041] The formula for converting the compatibility score to the initial edge weight after temperature-based softmax normalization is as follows:

[0042] in, This represents the value of node i as the center at time t, and its neighborhood. The normalized initial edge weights calculated for node j in the equation are as follows: ; It is the temperature coefficient, and >0; k represents the index of a local node within the subgraph. Let represent the neighborhood set of node i, and let represent the compatibility score between node i and another node k at time t.

[0043] In this embodiment of the invention, the temporal attention network unit receives the multimodal fusion feature sequence output by the cross-modal graph neural network unit, and performs self-attention calculation on the multimodal fusion feature sequence. Through the self-attention mechanism, differentiated temporal dimension attention weights are assigned to the multimodal fusion features at different time steps in the feature sequence.

[0044] In a preferred embodiment of the invention, the confidence score from the modal confidence evaluation subunit is coupled with the initial edge weight value obtained in the previous step to achieve a smooth amplification / attenuation of the edge weight by the confidence. A strategy of multiplying the edge weight by a function of the confidence of the participating endpoints and then renormalizing it ensures that when the confidence of a certain modality decreases, the edge weights connecting that modality automatically weaken, while the edge weights between it and other high-confidence modalities are relatively strengthened, thereby achieving online complementarity between modalities.

[0045] For the confidence-coupled part, let the confidence score of mode m be... The confidence coupling function is defined as follows:

[0046] in, This represents the confidence score of mode i at time t. This represents the confidence score of mode j at time t. This represents the amplification factor based on the confidence levels at both endpoints. This represents the confidence level sensitivity coefficient, and ≥0.

[0047] The formula for correcting and normalizing the initial edge weights is as follows:

[0048] in, This represents the intermediate edge weights after confidence coupling and renormalization. This represents the value of node i as the center at time t, and its neighborhood. The initial normalized edge weights are calculated from node k in the equation.

[0049] In a preferred embodiment of the present invention, to avoid blind weight switching caused solely by confidence levels, the dynamic edge weight update subunit also calculates the complementarity score or causal contribution of modality pairs from training statistics or online samples (e.g., estimating the improvement / compensation effect of other modalities on task performance when a certain modality is down / missing through counterfactual intervention), and uses this complementarity score as a fine-tuning factor for the edge weights. This ensures that in scenarios where speech degrades but visual pose combinations have significant complementarity for emotion judgment, the edge weights between visual poses are preferentially enhanced, guaranteeing the stability of the overall judgment performance. The final edge weight calculation formula after combining the complementarity score is as follows:

[0050] in, This represents the final edge weight after combining the complementarity scores. This indicates the complementarity fine-tuning intensity hyperparameter, and ≥0, This represents the complementarity score of node j to node i at time step t. This represents the complementarity score of node k to node i at time step t.

[0051] In a preferred embodiment of the invention, the actual edge weights are obtained through dual-scale weight maintenance. A dual-track update strategy is employed, using short-term fast-response weights (for quickly responding to sudden noise or short-term missing data) and long-term steady-state weights (for preserving historical robust relationships). These weights are maintained using exponential moving averages (EMAs) with different decay coefficients, and the actual edge weights for message passing are generated in a weighted manner. This ensures rapid response to sudden events while avoiding long-term model instability due to short-term anomalies. The dual-scale weight maintenance strategy maintains fast weights. With slow weights Its exponential moving average (EMA) update formula is as follows:

[0052]

[0053] in, , The weighted components are calculated at time step t using fast / slow exponential moving averages, respectively. , Both are EMA attenuation coefficients, and , , These represent the fast and slow edge weight components stored at the previous time step t-1, respectively.

[0054] The formula for coupling the fast weights and slow weights into the actual edge weights used for information transmission is as follows:

[0055] in, This represents the actual edge weights ultimately used for graph convolution message passing. Represents the weighted composite coefficient, and .

[0056] It should be noted that the actual edge weights are used for graph convolution message passing, and the formula for weighted neighborhood aggregation is as follows:

[0057] in, Let be the intermediate representation of node i at time t+1 after neighborhood weighted aggregation and activation. U is the activation function, and U represents the message linear transformation matrix (a learnable parameter). This represents the actual edge weights ultimately used for graph convolution message passing. This represents the embedding representation vector of subgraph node i at time step t.

[0058] In this embodiment of the invention, during the model training phase, the cross-modal contrastive learning unit constructs positive sample pairs and negative sample pairs based on each modal feature. It uses a preset contrastive loss function to maximize the similarity of modal features corresponding to the positive sample pairs and minimize the similarity of modal features corresponding to the negative sample pairs.

[0059] In a preferred embodiment of the present invention, during the training process, in addition to the conventional downstream emotion recognition loss, regularization terms are introduced to guide edge weight learning. These include complementary consistency loss (encouraging the allocation of larger edge weights to modality pairs with collaborative discriminative capabilities), sparse / structure regularization (promoting the interpretability and computational feasibility of the edge weight matrix), and confidence consistency penalty (imposing stronger constraints on edge weights related to low-confidence modalities), thereby improving the interpretability, generalization, and robustness of the edge weights. The total loss function used in the training phase is the sum of the downstream task loss and several regularization terms, calculated as follows:

[0060] Where L represents the total training loss, This represents the loss of the main task (emotion recognition). This represents complementary consistency loss, used to encourage the allocation of larger edge weights on modality pairs with cooperative discriminative capabilities. This represents a sparse or structure regularization term, used to improve the interpretability and computational efficiency of the edge weight matrix. This represents the confidence consistency loss. This represents the preset contrast loss function. , , , All represent weights.

[0061] It should be noted that the total training loss is used to optimize the training phase of cross-modal graph neural network units and their sub-units. By minimizing this loss function, the downstream emotion recognition accuracy, modal complementarity, edge weight sparsity and structural rationality, as well as confidence consistency constraints can be guaranteed at the same time. This guides the edge weight matrix and node representation to learn representations that are more in line with the modal complementarity rules during the parameter update process.

[0062] In this embodiment of the invention, the sequence smoothing unit receives the emotion recognition tag sequence output by the temporal attention network unit, and performs correlation constraints on the emotion recognition tag sequences of adjacent time steps through a preset sequence smoothing model, and outputs the current emotion tag and emotion intensity.

[0063] Specifically, to model the changes in emotions over time, the system introduces a Conditional Random Field (CRF) layer to smooth the output sequence and avoid abrupt changes in emotional states. Furthermore, an emotional memory network is designed to accumulate and analyze historical emotions and predict future emotional trends. For example, when the system detects a gradual increase in user anxiety, it triggers an early warning mechanism, prompting the digital human to adopt a more moderate interaction strategy. Unlike traditional static classification, this system not only focuses on instantaneous emotions but also achieves dynamic prediction, providing forward-looking support for subsequent interactions.

[0064] Furthermore, based on the current sentiment label, sentiment intensity, and historical sentiment trajectory, a composite response strategy is generated. This includes: using an sentiment memory network to model the user's historical sentiment trajectory and predict the user's sentiment state transition trend; using a candidate strategy generator, based on the predicted sentiment state transition trend and the current sentiment label, using a sequence-to-sequence neural network with an attention mechanism to generate multiple candidate composite response strategies; and using a policy planner, evaluating and selecting the candidate composite response strategies through a value function-based reinforcement learning method to obtain the composite response strategy. The composite response strategy includes a sentiment response intensity parameter, which is positively correlated with the sentiment intensity output by the cross-modal fusion and sentiment recognition module.

[0065] Specifically, after recognizing the user's emotions, the system enters the response planning phase. It employs "planned response generation" rather than direct template-based output. Specifically, it first models the user's emotional trajectory using an emotional memory network and generates a set of candidate response strategies based on the contextual dialogue. For example, when a user exhibits anxiety, candidate strategies might include "explanation + comfort" or "distracting attention"; when a user is confused, candidate strategies might be "detailed explanation + encouragement." Subsequently, the system uses a strategy planner trained through reinforcement learning to select the optimal strategy from the candidate set based on the emotional state, historical trajectory, and task objective. The planner evaluates the impact of different strategies on the interaction effect, such as "whether it reduces user anxiety," and uses this as a reward signal to continuously optimize. Unlike the end-to-end generation of GAN or VAE-like methods, this module emphasizes a hierarchical logic of "emotional state—strategy selection—action output," making the response more controllable and personalized. In psychological counseling scenarios, this method can significantly enhance the empathy of digital humans, increasing user trust and engagement.

[0066] In a preferred embodiment of the present invention, the strategy planner uses interaction effect indicators as reward signals. These indicators include the degree of improvement in the user's emotional state and the increase in user engagement. The degree of improvement in the user's emotional state is defined as "the difference in emotional intensity within 3 seconds before and after strategy execution." For example, if the anxiety intensity was 0.8 before executing the "comfort strategy" and 0.5 after, the difference is 0.3. A difference ≥ 0.2 is recorded as a positive reward (+1), and < 0 is recorded as a negative reward (-1). The increase in user engagement is defined as "the change in engagement indicators within 5 seconds before and after strategy execution," specifically including: Voice participation: Speech duration increased by ≥2 seconds (+0.5); Visual engagement: The percentage of time spent in eye contact increased by ≥10% (+0.3); User engagement: The number of times users click the interactive button increases by ≥1 (+0.2); Total participation improvement value = sum of all sub-indicators, ≥0.5 is recorded as positive reward (+1), <0 is recorded as negative reward (-1).

[0067] Reward signal = Emotional improvement reward × 0.6 + Participation reward × 0.4 (weights are configurable).

[0068] In this embodiment of the invention, the digital human rendering module includes: a speech synthesis unit, which decodes the composite response strategy using a sequence-to-sequence neural network based on the Tacotron architecture to generate a speech control signal; a facial expression generation unit, which generates a facial expression control signal based on the deformation parameterization model of the BlendShape basic expression component and combined with attention weights associated with the current emotion tag; an action generation unit, which constructs a graph structure with human skeleton joints as nodes and uses a graph convolutional network to predict the joint position trajectory at the next moment under the constraint of the composite response strategy to generate an action control signal; and a timing synchronization unit, which registers the speech control signal, facial expression control signal, and action control signal using a dynamic time warping algorithm and compensates for misaligned parts using a timestamp interpolation method.

[0069] Specifically, the planned responses are mapped to the digital human's actual performance through the rendering module. The speech component utilizes a Tacotron-like emotional speech synthesizer, combined with intonation control and prosody adjustment, to imbue the output speech with emotional color, such as slowing down the speech rate when anxious and raising the tone when encouraging. The facial expressions component uses BlendShape combined with attention-weighted control to achieve subtle changes in the corners of the mouth, eyebrows, and eyes; the motion component is driven by skeletal binding and inverse kinematics (IK) algorithms to achieve natural gestures and body postures. For example, when the user shows tension, the digital human will nod with a soft tone; when the user shows confusion, it will be accompanied by explanatory gestures and focused eye contact. To ensure coordination, the system introduces a multi-channel timing synchronization mechanism to strictly align speech, facial expressions, and motion, with a time error of less than 50ms. The final digital human not only responds at the linguistic level but also exhibits natural and believable interactive performances in non-verbal signals, allowing the user to feel that their emotions are truly understood and responded to.

[0070] The present invention also provides a digital human interaction method based on multimodal emotion recognition, for implementing a digital human interaction system based on multimodal emotion recognition. The system includes a control module, the control module including a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the digital human interaction system based on multimodal emotion recognition.

[0071] This invention provides a storage medium storing a program that, when executed by a processor, implements the digital human interaction system based on multimodal emotion recognition.

[0072] This invention provides a processor for running a program, wherein the program executes the digital human interaction system based on multimodal emotion recognition.

[0073] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a digital human interaction system based on multimodal emotion recognition. The device described herein can be a server, PC, tablet, mobile phone, etc.

[0074] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing a digital human interaction system based on multimodal emotion recognition.

[0075] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0080] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0081] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0082] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0083] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A digital human interaction system based on multimodal emotion recognition, characterized in that, include: A multimodal sensing module is used to collect multimodal data and preprocess the multimodal data to generate a standardized data stream; The cross-modal fusion and emotion recognition module is used to receive the standardized data stream, perform interactive modeling of multimodal features, and output the current emotion label and emotion intensity. The response planning module is used to generate a composite response strategy based on the current emotion tag, the emotion intensity, and the historical emotion trajectory. The digital human rendering module is used to map the composite response strategy into control signals corresponding to voice, facial expressions and actions, respectively. The control signals drive the digital human to perform corresponding voice output, facial expression changes and body movements to achieve interaction.

2. The digital human interaction system based on multimodal emotion recognition according to claim 1, characterized in that, The preprocessing of the multimodal data to generate a standardized data stream includes: Denoising is performed on the speech signal and Mel spectrum and prosodic features are extracted; Semantic vectorization of text based on a pre-trained language representation model; Visual signals are extracted using a 3D convolutional network to capture facial micro-expressions and facial dynamic features. Limb movement features are extracted from posture signals using graph convolutional networks; Physiological features are extracted from physiological signals through one-dimensional convolution and temporal modeling. The modal features mentioned above are aligned and standardized in the time dimension to generate a standardized data stream.

3. The digital human interaction system based on multimodal emotion recognition according to claim 1, characterized in that, The cross-modal fusion and emotion recognition module includes: Cross-modal graphical neural network unit, cross-modal contrastive learning unit, temporal attention network unit, and sequence smoothing unit; The cross-modal graph neural network unit receives modal features output by the multimodal perception module and maps each modal feature to an independent child node in the graph structure. By performing a preset graph convolution operation and calling the configured learnable edge weight matrix, the relationship between each independent child node is modeled. During the model training phase, the cross-modal contrastive learning unit constructs positive and negative sample pairs based on each modal feature. It uses a preset contrastive loss function to maximize the similarity of the modal features corresponding to the positive sample pairs and minimize the similarity of the modal features corresponding to the negative sample pairs. The temporal attention network unit receives the multimodal fusion feature sequence output by the cross-modal graph neural network unit, and performs self-attention calculation on the multimodal fusion feature sequence, assigning differentiated temporal dimension attention weights to the multimodal fusion features at different time steps in the feature sequence through the self-attention mechanism; The sequence smoothing unit receives the emotion recognition label sequence output by the temporal attention network unit, and performs correlation constraints on the emotion recognition label sequences of adjacent time steps through a preset sequence smoothing model, and outputs the current emotion label and emotion intensity.

4. The digital human interaction system based on multimodal emotion recognition according to claim 3, characterized in that, The cross-modal graphical neural network unit includes: A modal confidence evaluation subunit calculates the confidence score of each modality based on the short-time consistency index, the confidence score of the pre-trained encoder output, and / or the signal-to-noise ratio. The dynamic edge weight update subunit performs adaptive regularization on the learnable edge weight matrix in response to the following conditions: the confidence score is lower than a predetermined threshold, or the noise of the modal feature exceeds a predetermined noise threshold, or the condition of missing modal feature is met. This reduces the weight value of the edge corresponding to the modal feature in the graph structure.

5. The digital human interaction system based on multimodal emotion recognition according to claim 4, characterized in that, The adaptive regularization of the learnable edge weight matrix includes obtaining the actual edge weights by maintaining dual-scale weights.

6. The digital human interaction system based on multimodal emotion recognition according to claim 1, characterized in that, The step of generating a composite response strategy based on the current sentiment tag, the sentiment intensity, and the historical sentiment trajectory includes: By using emotional memory networks, we can model users' historical emotional trajectories and predict the trends in the shifts of users' emotional states. Using a candidate policy generator, multiple candidate composite response policies are generated based on the predicted shift trend of user sentiment state and current sentiment label, and a sequence-to-sequence neural network with attention mechanism is employed. The candidate composite response strategies are evaluated and selected using a policy planner and a value function-based reinforcement learning method to obtain composite response strategies.

7. The digital human interaction system based on multimodal emotion recognition according to claim 6, characterized in that, The strategy planner uses interaction performance metrics as reward signals, including the degree of improvement in user emotional state and the increase in user engagement.

8. The digital human interaction system based on multimodal emotion recognition according to claim 1, characterized in that, The composite response strategy includes an emotion response intensity parameter, which is positively correlated with the emotion intensity output by the cross-modal fusion and emotion recognition module.

9. The digital human interaction system based on multimodal emotion recognition according to claim 1, characterized in that, The digital human rendering module includes: The speech synthesis unit decodes the composite response strategy using a sequence-to-sequence neural network based on the Tacotron architecture to generate speech control signals; The facial expression generation unit generates facial expression control signals based on the deformation parameterization model of the BlendShape basic expression component and combined with the attention weights associated with the current emotion tag. The motion generation unit constructs a graph structure with human skeletal joints as nodes. It uses a graph convolutional network to predict the joint position trajectory at the next moment under the constraint of a composite response strategy, and generates motion control signals. The timing synchronization unit registers the voice control signal, facial expression control signal, and motion control signal using a dynamic time warping algorithm, and compensates for misaligned parts using a timestamp interpolation method.

10. A digital human interaction method based on multimodal emotion recognition, characterized in that, The digital human interaction method based on multimodal emotion recognition includes: Collect multimodal data and preprocess the multimodal data to generate a standardized data stream; Receive the standardized data stream, perform interactive modeling on the multimodal features, and output the current sentiment label and sentiment intensity; A composite response strategy is generated based on the current sentiment tag, the sentiment intensity, and the historical sentiment trajectory. The composite response strategy is mapped to control signals corresponding to speech, facial expressions, and actions, respectively. The control signals drive the digital human to perform corresponding speech output, facial expression changes, and body movements to achieve interaction.

Images