Digital human interaction system and method based on multi-modal emotion recognition

Abstract

The embodiment of the application provides a digital human interaction system and method based on multi-modal emotion recognition, and belongs to the technical field of digital human interaction; the system comprises a multi-modal perception module, which is used for collecting multi-modal data, pre-processing the multi-modal data, and generating standardized data flow; a cross-modal fusion and emotion recognition module, which is used for interactive modeling of multi-modal features, and outputs current emotion labels and emotion intensity; a reaction planning module, which is used for generating a composite reaction strategy; and a digital human rendering module, which is used for mapping the composite reaction strategy into control signals corresponding to voice, facial expression and action respectively, and driving the digital human to perform corresponding voice output, facial expression change and limb action through the control signals to realize interaction. Through cross-modal graph neural network and contrast learning, the multi-modal data is deeply fused, the weight is dynamically adjusted in combination with the modal confidence, and the emotion recognition accuracy and robustness are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital human interaction, in particular to a digital human interaction system and method based on multi-modal emotion recognition. BACKGROUND

[0002] In recent years, digital humans have gradually moved out of the entertainment field and entered high-demand scenarios such as medical treatment, government affairs, and education. However, the existing digital human interaction generally has the problem of "lack of emotion", which makes it difficult for the interaction performance to gain the trust of users. For example, in psychological counseling, if the digital human only answers with mechanical voice and cannot capture the emotions such as anxiety, depression, or excitement of the user, the service effect will be greatly reduced. Traditional emotion recognition methods often rely on a single modality (such as speech emotion recognition or text emotion analysis), lack of integration of multi-source emotional clues, and are difficult to accurately reflect complex emotional states. On the other hand, most of the current multi-modal methods use simple feature splicing or early fusion, which cannot fully exploit the temporal relationship and complementarity between different modalities, resulting in limited recognition performance. At the same time, the reaction generation of digital humans is mostly based on fixed rules or preset templates, which lacks dynamic adaptation to user emotions and is difficult to achieve "different for different people" in the interaction process. Therefore, there is an urgent need for a new system that can deeply integrate multi-modal information, dynamically model emotional state transitions, and drive digital humans to generate emotion-adaptive reactions. The solution proposed in the present application combines graph neural networks, cross-modal attention mechanisms, and emotion memory networks, not only improving the accuracy and robustness of emotion recognition, but also enabling digital humans to have emotion-driven interaction capabilities, significantly enhancing user experience. SUMMARY

[0003] The purpose of the present application is to provide a digital human interaction system and method based on multi-modal emotion recognition, which is used to drive digital humans to generate natural interaction with empathy ability through multi-modal emotion recognition and dynamic reaction planning, and is applied to scenarios such as psychological counseling and intelligent customer service.

[0004] In order to achieve the above-mentioned purpose, the present application provides a digital human interaction system based on multi-modal emotion recognition, comprising: a multi-modal perception module for collecting multi-modal data and preprocessing the multi-modal data to generate standardized data streams; a cross-modal fusion and emotion recognition module for receiving the standardized data streams and modeling multi-modal features, outputting current emotion labels and emotion intensity; a reaction planning module for generating a composite reaction strategy according to the current emotion labels, the emotion intensity and the historical emotional trajectory; a digital human rendering module for mapping the composite reaction strategy into control signals corresponding to voice, facial expression and action respectively, and driving the digital human to execute corresponding voice output, facial expression change and body movement to realize interaction through the control signals.

[0005] Optionally, the preprocessing of the multi-modal data to generate a standardized data stream comprises: performing denoising processing on the voice signal and extracting mel-frequency spectrum and prosody features; performing semantic vectorization processing on the text based on a pre-trained language representation model; extracting facial micro-expression and facial dynamic features from the visual signal through a three-dimensional convolution network; extracting limb action features from the posture signal based on a graph convolution network; extracting physiological features from the physiological signal through one-dimensional convolution and time series modeling; aligning and standardizing the features of each modality in the time dimension to generate a standardized data stream; aligning and standardizing the features of each modality in the time dimension to generate a standardized data stream.

[0006] Optionally, the cross-modal fusion and emotion recognition module comprises: a cross-modal graph neural network unit, a cross-modal contrast learning unit, a time sequence attention network unit, and a sequence smoothing unit; the cross-modal graph neural network unit receives the features of each modality output by the multi-modal perception module, and maps the features of each modality into independent sub-nodes in a graph structure, respectively, models the association relationship between the independent sub-nodes by performing a preset graph convolution operation and calling a configured learnable edge weight matrix; the cross-modal contrast learning unit constructs positive sample pairs and negative sample pairs based on the features of each modality in the model training stage, and uses a preset contrast loss function to perform a maximization constraint on the similarity of the modality features corresponding to the positive sample pairs, and simultaneously performs a minimization constraint on the similarity of the modality features corresponding to the negative sample pairs; the time sequence attention network unit receives the multi-modal fusion feature sequence output by the cross-modal graph neural network unit, and performs self-attention calculation on the multi-modal fusion feature sequence, and assigns different time dimension attention weights to the multi-modal fusion features at different time steps in the feature sequence through the self-attention mechanism; the sequence smoothing unit receives the emotion recognition label sequence output by the time sequence attention network unit, and performs correlation constraint on the emotion recognition label sequences at adjacent time steps through a preset sequence smoothing model, and outputs the current emotion label and emotion intensity.

[0007] Optionally, the cross-modal graph neural network unit comprises: a modality confidence evaluation sub-unit that calculates a confidence score of each modality based on a short-term consistency indicator, a confidence output by a pre-trained encoder, and / or a signal-to-noise ratio; and a dynamic edge weight updating sub-unit that performs adaptive regularization on the learnable edge weight matrix to weaken the weight value of the edge corresponding to the modality feature in the graph structure in response to the confidence score being lower than a predetermined threshold, or in response to a condition that the noise of the modality feature exceeds a predetermined noise threshold being detected, or a condition that the modality feature is missing being detected.

[0008] Optionally, the adaptive regularization on the learnable edge weight matrix comprises: obtaining an actual edge weight through double-scale weight maintenance.

[0009] Optionally, the generating the composite response strategy according to the current emotional label, the emotional intensity and the historical emotional trajectory comprises: modeling the historical emotional trajectory of the user by using an emotional memory network to predict a transfer trend of the emotional state of the user; generating a plurality of candidate composite response strategies by using a candidate strategy generator based on the predicted transfer trend of the emotional state of the user and the current emotional label by using a sequence-to-sequence neural network with an attention mechanism; and obtaining the composite response strategy by using a strategy planner to evaluate and select the candidate composite response strategies by a reinforcement learning method based on a value function.

[0010] Optionally, the strategy planner takes an interaction effect index as a reward signal, and the interaction effect index comprises a user emotional state improvement degree and a user participation degree improvement value.

[0011] Optionally, the composite response strategy comprises an emotional response intensity parameter, and the emotional response intensity parameter is in a positive correlation mapping relationship with the emotional intensity output by the cross-modal fusion and emotional recognition module.

[0012] Optionally, the digital human rendering module comprises: a speech synthesis unit configured to decode the composite response strategy by using a sequence-to-sequence neural network based on a Tacotron architecture to generate a speech control signal; a facial expression generation unit configured to generate a facial expression control signal based on a deformation parameterization model of a BlendShape basic expression component in combination with an attention weight associated with the current emotional label; a motion generation unit configured to construct a graph structure with human skeleton joints as nodes, and to predict a joint position trajectory of a next time point under the constraint of the composite response strategy by using a graph convolution network to generate a motion control signal; and a timing synchronization unit configured to register the speech control signal, the facial expression control signal and the motion control signal by using a dynamic time warping algorithm, and to compensate for unaligned parts by using a time stamp interpolation method.

[0013] In another aspect, the present application provides a digital human interaction method based on multi-modal emotional recognition, comprising: collecting multi-modal data, and preprocessing the multi-modal data to generate standardized data streams; receiving the standardized data streams, and modeling multi-modal features to output a current emotional label and an emotional intensity; generating a composite response strategy according to the current emotional label, the emotional intensity and a historical emotional trajectory; mapping the composite response strategy into control signals corresponding to speech, facial expression and motion respectively, and driving a digital human to perform corresponding speech output, facial expression change and limb motion to realize interaction by using the control signals.

[0014] The technical scheme has the advantages that the multi-modal data is deeply fused through the cross-modal graph neural network and the contrast learning, the weight is dynamically adjusted in combination with the modal confidence, the emotion recognition accuracy and robustness are improved, the dynamic compound reaction strategy is generated relying on the emotion memory network and the reinforcement learning, the limitation of the fixed template is avoided, the empathetic interaction that is different for different people is realized, and the user trust and the interactive experience are significantly improved.

[0015] Other features and advantages of the present application will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, and are used together with the following detailed description to explain the application, but do not constitute a limitation on the application. In the drawings:

[0017] Figure 1 is a digital human interaction system diagram.

[0018] Figure 2 is a digital human interaction flowchart based on multi-modal emotion recognition.

[0019] Figure 3 is a multi-modal data preprocessing flowchart. DETAILED DESCRIPTION

[0020] The following will be described in detail with reference to the accompanying Figure 1 - the accompanying Figure 3 The specific implementation of the embodiment of the present application is described in detail. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present application, and is not used to limit the embodiment of the present application.

[0021] It should be noted that the acquisition, transmission, storage, use, processing and the like of data in the technical scheme of the present application comply with the relevant provisions of national laws and regulations. In the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models and the like, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical scheme of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0022] The present inventors found in the process of implementing the present application that the prior art has the defects of lack of digital human emotion, shallow multi-modal fusion, fixed reaction, poor user trust and experience.

[0023] Embodiment 1

[0024] Reference Figures 1-3For the first embodiment of the present application, the embodiment provides a digital human interaction system based on multi-modal emotion recognition, which can include: a multi-modal perception module for collecting multi-modal data and pre-processing the multi-modal data to generate standardized data streams; a cross-modal fusion and emotion recognition module for receiving the standardized data streams and modeling multi-modal features for interaction, outputting current emotion labels and emotion intensity; a reaction planning module for generating a composite reaction strategy based on the current emotion label, emotion intensity and historical emotional trajectory; a digital human rendering module for mapping the composite reaction strategy to control signals corresponding to voice, facial expression and action respectively, and driving the digital human to perform corresponding voice output, facial expression change and body movement through the control signals to realize interaction.

[0025] In the embodiment of the present application, the digital human interaction system first collects real-time interaction data of the user through multi-modal sensing devices. Referring to Figure 3 The pre-processing of the multi-modal data to generate standardized data streams can include:

[0026] S110: Perform denoising processing on the voice signal and extract Mel spectrum and prosody features.

[0027] Wherein, the voice signal collects the original audio through a high-sensitivity microphone, uses endpoint detection to eliminate silence, and uses spectral subtraction and Wiener filtering for voice enhancement to remove noise, and then extracts the Mel spectrum feature through a convolutional neural network (CNN), and analyzes the fundamental frequency, energy and other prosody information.

[0028] S120: Perform semantic vectorization processing on the text based on a pre-trained language representation model.

[0029] Wherein, the text data is derived from the dialogue record with the digital human, which is transcribed in real time by a natural language processing engine and annotated with a timestamp. A pre-trained language model such as RoBERTa is used for deep semantic coding to accurately capture the sentiment polarity and potential emotional clues in the sentence.

[0030] S130: Extract facial micro-expression and facial dynamic features from visual signals through a three-dimensional convolutional network.

[0031] Wherein, the visual signal is first captured by a high-definition camera, and then the facial region is obtained by face key point detection, and dynamic micro-expression and gaze changes are captured with the help of optical flow tracking; the visual modality uses a 3D convolutional network to extract the above facial micro-expression and action features.

[0032] S140: Extract limb action features based on a graph convolutional network.

[0033] The posture signal is obtained by combining a posture recognition module and a related collection device (such as a camera for capturing body movement) to extract human skeleton point data (including head tilt, gesture action and body posture), and further extracting a limb movement mode through human skeleton point detection and a graph convolutional network (GCN) to form a posture feature that can be used for emotion recognition.

[0034] In S150, a physiological signal is subjected to one-dimensional convolution and time sequence modeling to extract a physiological feature.

[0035] The physiological signal is collected by a wearable device such as a smart bracelet, and physiological data such as heart rate and skin electric response can be obtained to provide emotional clues at the physiological level for subsequent emotion recognition.

[0036] In S160, the modal features are aligned and standardized in the time dimension to generate a standardized data stream.

[0037] All modal data are subjected to multi-stream synchronization through a unified timestamp management mechanism, the frame rate inconsistency problem is solved by using an interpolation alignment algorithm, and standardization and dimension reduction are used at the feature layer to ensure the consistency of the input, so that a standardized data stream is obtained. Rich and high-quality basic information is provided for emotion recognition.

[0038] In the embodiment of the present application, the modal fusion and emotion recognition module comprises a cross-modal graph neural network unit, a cross-modal contrast learning unit, a time sequence attention network unit and a sequence smoothing unit.

[0039] In the embodiment of the present application, the cross-modal graph neural network unit receives the modal features output by the multi-modal perception module, and maps each modal feature into an independent sub-node in the graph structure, and models the association relationship between each independent sub-node by performing a predetermined graph convolution operation and calling a configured learnable edge weight matrix.

[0040] In the preferred embodiment of the present application, the cross-modal graph neural network unit comprises a modal confidence evaluation sub-unit and a dynamic edge weight updating sub-unit. The modal confidence evaluation sub-unit calculates the confidence score of each modal based on a short-term consistency index (a 3-second window can be used), the confidence and / or signal-to-noise ratio output by the pre-trained encoder (an adaptive learning weight can be used). The dynamic edge weight updating sub-unit performs adaptive regularization on the learnable edge weight matrix to weaken the weight value of the edge corresponding to the modal feature in the graph structure when the confidence score is lower than a predetermined threshold, or when the noise of the modal feature exceeds a predetermined noise threshold, or when the condition of missing modal feature is detected.

[0041] In the preferred embodiment of the present application, to achieve fine complementary and robust fusion at the modal level, the cross-modal graph neural network unit constructs each modality as a subgraph node in the graph structure, i.e., the speech, text, visual, posture, and physiological signal modalities correspond to independent subgraph nodes, respectively. Each subgraph node not only carries the time sequence feature vector of the modality, but also retains the local structure information within the modality (such as multi-region features of the visual modality, a set of joint nodes of the posture modality, multi-channel signals of the physiological modality, etc.), so as to express the internal semantic information of the modality and the cross-modal interface information at the node level at the same time. The association between the subgraph nodes is represented by a learnable edge weight matrix, which is dynamically updated in an adaptive mechanism during training and online inference, so as to realize the complementary behavior of automatically increasing the weight of the complementary modality (such as visual or posture) when a certain modality (such as speech) degrades or is missing.

[0042] In the preferred embodiment of the present application, the adaptive mechanism dynamic update includes: for any two subgraph nodes i, j, first calculate the compatibility score by a lightweight compatibility mapping, and the calculation formula of the compatibility score is as follows:

[0043]

[0044] wherein, represents the compatibility score of node i and node j at time t, represents a nonlinear mapping function of compatibility, and both represent a learnable linear projection matrix (weight matrix), and represent the embedding representation (vector) of node i, j at time t, represents an element-wise product, represents a bias term.

[0045] The compatibility score is temperatureed and normalized by softmax to obtain the edge weight initial value, and the formula is as follows:

[0046]

[0047] wherein, represents the normalized edge weight initial value calculated for node j in the neighborhood of node i at time t, and ; is a temperature coefficient, and > 0; k represents a local node index within the subgraph, represents the neighborhood set of node i, and represents the compatibility score between node i and another node k at time t.

[0048] ​In the embodiment of the present application, the timing attention network unit receives the multi-modal fusion feature sequence output by the cross-modal graph neural network unit, and performs self-attention calculation on the multi-modal fusion feature sequence, and assigns different time dimension attention weights to the multi-modal fusion features at different time steps in the feature sequence through the self-attention mechanism.

[0049] In the preferred embodiment of the present application, the confidence score from the modal confidence evaluation subunit is coupled with the edge weight initial value obtained in the previous step to achieve smoothing amplification / attenuation of the confidence to the edge weight. The strategy of multiplying the edge weight by a function of the confidence of the participating endpoints and then re-normalizing is adopted, so that when the confidence of a certain modal decreases, the edge weight connected to this modal is automatically weakened, while the edge weight between other high-confidence modes is relatively enhanced, thereby realizing online complementarity between modes.

[0050] For the confidence coupling part, let the confidence score of modal m be The confidence coupling function is defined as follows:

[0051]

[0052] wherein, denotes the confidence score of modal i at time t, denotes the confidence score of modal j at time t, denotes an amplification factor based on the confidence of two endpoints, denotes a confidence-sensitive coefficient, and ≥0.

[0053] The formula for modifying the initial edge weight and normalizing is as follows:

[0054]

[0055] wherein, denotes the intermediate edge weight after confidence coupling and re-normalization, denotes the initial normalized edge weight calculated for node k in the neighborhood of node i at time t.

[0056] In the preferred embodiment of the present application, in order to avoid blind weight switching caused by confidence alone, the dynamic edge weight updating subunit also calculates the complementarity score or causal contribution degree of the modal pair (for example, by counterfactual intervention to estimate the improvement / compensation effect of other modal pairs on task performance when a certain modal is lowered / missing) from the training period statistics or online samples, and uses the complementarity score as a fine-tuning factor for the edge weight, so that in the scene where the voice is degraded but the visual pose combination has significant complementarity to emotion determination, the edge weight between the visual poses is preferentially enhanced, ensuring the stability of the overall determination performance. The calculation formula of the final edge weight combined with the complementarity score is as follows:​

[0057]

[0058] wherein, denotes the final edge weight after combining the complementarity score, denotes the complementarity fine-tuning strength hyperparameter, and ≥ 0, denotes the complementarity score of node j to node i at time step t, denotes the complementarity score of node k to node i at time step t.

[0059] In the preferred embodiment of the present application, the actual edge weight is maintained by dual-scale weight maintenance. A dual-track updating strategy of short-term fast response weight (for fast response to sudden noise or short-term missing) and long-term steady-state weight (for preserving historical robust relationships) is adopted, which is maintained by exponential moving average (EMA) with different decay coefficients respectively, and the actual edge weight for message passing is generated in a weighted manner, so as to ensure fast response to sudden events while avoiding long-term model instability caused by short-term abnormalities. The dual-scale weight maintenance strategy, i.e. maintaining fast weight and slow weight The exponential moving average (EMA) update formula is as follows:

[0060]

[0061]

[0062] wherein, , are the weight components calculated by fast / slow exponential moving average at time step t respectively, , are EMA decay coefficients, and , , denote the fast / slow edge weight components saved at the previous time step t-1 respectively.

[0063] The formula for coupling the fast weight and the slow weight into the actual edge weight for information transmission is as follows:

[0064]

[0065] wherein, denotes the actual edge weight for graph convolution message passing finally, denotes the weight synthesis coefficient, and .

[0066] It should be noted that the actual edge weight is used for graph convolution message passing, and the formula for weighted neighborhood aggregation is as follows:

[0067]

[0068] wherein, is the intermediate representation of node i after neighborhood weighted aggregation and activation at time t+1, is the activation function, and U represents the message linear transformation matrix (learnable parameters), represents the actual edge weight used for graph convolution message passing, represents the embedding representation vector of subgraph node i at time step t.

[0069] In the embodiment of the present application, the cross-modal contrast learning unit constructs positive sample pairs and negative sample pairs based on the features of each modality in the model training stage, and uses a preset contrast loss function to perform a maximum constraint on the similarity of the modality features corresponding to the positive sample pairs, and simultaneously performs a minimum constraint on the similarity of the modality features corresponding to the negative sample pairs.

[0070] In the preferred embodiment of the present application, in addition to the conventional downstream sentiment recognition loss in the training process, a regular term for guiding edge weight learning is introduced, including a complementary consistency loss (encouraging larger edge weights to be assigned on modality pairs with collaborative discrimination ability), a sparse / structural regularity (promoting the edge weight matrix to have interpretability and computational realizability), and a confidence consistency penalty (applying stronger constraints on low-confidence modality-related edge weights), so as to improve the interpretability, generalizability and robustness of the edge weights. The total loss function used in the training stage is the sum of the downstream task loss and several regular terms, and the calculation formula is as follows:

[0071]

[0072] wherein, L represents the total training loss, represents the main task loss (sentiment recognition), represents the complementary consistency loss, which is used to encourage larger edge weights to be assigned on modality pairs with collaborative discrimination ability, represents the sparse or structural regular term, which is used to improve the interpretability and computational efficiency of the edge weight matrix, represents the confidence consistency loss, represents the preset contrast loss function, 、 、 、 all represent weights.

[0073] It should be noted that the total training loss is used for the training stage optimization of the cross-modal graph neural network unit and its sub-units. By minimizing the loss function, the downstream sentiment recognition accuracy, modality complementarity, edge weight sparsity, structural rationality and confidence consistency constraints can be ensured at the same time, so as to guide the edge weight matrix and node representation learning to be more consistent with the modality complementary rule in the parameter updating process.

[0074] In the embodiment of the application, the sequence smoothing unit receives the sentiment recognition label sequence output by the time sequence attention network unit, and performs correlation constraint on the sentiment recognition label sequences of adjacent time steps through a preset sequence smoothing model to output the current sentiment label and the sentiment intensity.

[0075] Specifically, in order to realize modeling of the change of emotion over time, the system introduces a conditional random field (CRF) layer to smooth the output sequence and avoid abrupt jumps in the sentiment state. In addition, a sentiment memory network is designed to cumulatively analyze historical emotions and predict future emotional trends. For example, when the system identifies that the user's anxiety emotion is gradually increasing, the system will trigger a warning mechanism to prompt the digital person to adopt a mitigating interaction strategy. Unlike traditional static classification, the recognition of the system not only focuses on instantaneous emotion, but also realizes dynamic prediction to provide forward-looking support for subsequent interaction.

[0076] Further, according to the current sentiment label, the sentiment intensity and the historical emotional trajectory, a composite response strategy is generated, including: modeling the historical emotional trajectory of the user by using the sentiment memory network to predict the transfer trend of the user's emotional state; using a candidate strategy generator to generate multiple candidate composite response strategies based on the predicted transfer trend of the user's emotional state and the current sentiment label by using a sequence-to-sequence neural network with an attention mechanism; using a strategy planner to evaluate and select the candidate composite response strategies by using a reinforcement learning method based on a value function to obtain the composite response strategy. The composite response strategy includes an emotional response intensity parameter, and the emotional response intensity parameter is in a positive correlation mapping relationship with the sentiment intensity output by the cross-modal fusion and sentiment recognition module.

[0077] Specifically, after identifying the user's emotion, the system enters the reaction planning stage. Instead of direct templating output, a "planned reaction generation" is adopted. Specifically, first, the user's emotional trajectory is modeled through the emotional memory network, and a set of candidate reaction strategies is generated in combination with the context dialogue content. For example, when the user shows anxiety, the candidate strategies may include "explanation + comfort" or "diverting attention"; when the user is confused, the candidate strategy may be "detailed explanation + encouragement". Subsequently, the system uses a policy planner trained by reinforcement learning to select the optimal strategy from the candidate set according to the emotional state, historical trajectory, and task goal. The planner will evaluate the impact of different strategies on the interaction effect, such as "whether to reduce the user's anxiety level", and use it as a reward signal for continuous optimization. Unlike the end-to-end generation of GAN or VAE methods, this module emphasizes the hierarchical logic of "emotional state - strategy selection - action output", making the reaction more controllable and personalized. In the psychological counseling scenario, this method can significantly enhance the digital human's empathy ability, and improve the user's trust and participation.

[0078] In the preferred embodiment of the present application, the policy planner uses the interaction effect index as the reward signal, which includes the user's emotional state improvement and the user's participation level improvement. The user's emotional state improvement is defined as the difference in emotional intensity within 3 seconds before and after the execution of the strategy. For example, the anxiety intensity before executing the "comfort strategy" is 0.8, and after executing the "comfort strategy", it is 0.5. The difference is 0.3, and the difference ≥0.2 is considered as a positive reward (+1), and <0 is considered as a negative reward (-1). The user's participation level improvement is defined as the change in participation index within 5 seconds before and after the execution of the strategy, which specifically includes:

[0079] Voice participation: speech duration increase ≥2 seconds (+0.5);

[0080] Visual participation: eye contact duration ratio increase ≥10% (+0.3);

[0081] Operation participation: user click interaction button number increase ≥1 time (+0.2);

[0082] Total participation level improvement = sum of each sub-index, ≥0.5 is considered as a positive reward (+1), and <0 is considered as a negative reward (-1).

[0083] Reward signal = emotional improvement reward × 0.6 + participation reward × 0.4 (weight can be configured).

[0084] In the embodiment of the present application, the digital human rendering module comprises: a speech synthesis unit, which decodes the composite reaction strategy through a sequence-to-sequence neural network based on a Tacotron architecture to generate a speech control signal; a facial expression generation unit, which generates a facial expression control signal based on a morphing parameterization model of a BlendShape basic expression component combined with attention weights associated with the current emotional label; an action generation unit, which constructs a graph structure with human skeleton joints as nodes, and predicts the joint position trajectory at the next moment under the constraint of the composite reaction strategy using a graph convolution network to generate an action control signal; and a timing synchronization unit, which registers the speech control signal, the facial expression control signal and the action control signal through a dynamic time warping algorithm, and compensates for the misaligned part through a timestamp interpolation method.

[0085] Specifically, the planned reaction is mapped to the specific performance of the digital human through the rendering module. The speech part uses a Tacotron-like emotional speech synthesizer combined with intonation control and prosody adjustment to make the output speech have emotional color, such as slowing down the speech speed when anxious and lifting the tone when encouraging. The expression part realizes delicate changes in the mouth, eyebrows and eyes through BlendShape combined with attention weighting control; the action part realizes natural gestures and body postures through bone binding and inverse kinematics (IK) algorithm driving. For example, when the user shows nervousness, the digital human will use a nodding action with a soft tone; when the user shows confusion, it will be accompanied by an explanatory gesture and a focused gaze. To ensure coordination, the system introduces a multi-channel timing synchronization mechanism to strictly align the speech, expression and action, with a time error of less than 50ms. The final presented digital human not only gives a response at the language level, but also shows natural and credible interactive performance on non-verbal signals, making the user feel that the emotion is truly understood and responded.

[0086] The present application also provides a digital human interaction method based on multi-modal emotion recognition, which is used to realize a digital human interaction system based on multi-modal emotion recognition. The system comprises a control module, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor. The processor executes the computer program to realize the digital human interaction system based on multi-modal emotion recognition.

[0087] The embodiment of the present application provides a storage medium having a program stored thereon, which is executed by a processor to realize the digital human interaction system based on multi-modal emotion recognition.

[0088] The embodiment of the present application provides a processor for running a program, wherein the program is executed to realize the digital human interaction system based on multi-modal emotion recognition.

[0089] The embodiment of the present application provides a device, the device comprises a processor, a memory and a program stored on the memory and executable on the processor, and the processor implements a digital human interaction system based on multi-modal emotion recognition when executing the program. The device herein can be a server, a PC, a PAD, a mobile phone and the like.

[0090] The present application also provides a computer program product adapted to execute the digital human interaction system based on multi-modal emotion recognition when executed on a data processing device.

[0091] Those skilled in the art should understand that the embodiments of the present application can provide a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage and the like) containing computer usable program codes.

[0092] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks. Figure 1 The function specified in one or more flows and / or blocks.

[0093] These computer program instructions can also be stored in a computer readable storage medium capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks. Figure 1 The function specified in one or more flows and / or blocks.

[0094] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks. Figure 1 The function specified in one or more flows and / or blocks.

[0095] In one typical arrangement, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0096] Memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores the information. Memory is an example of computer readable media.

[0097] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0098] It should also be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0099] The above merely provides an example of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A digital human interaction system based on multi-modal emotion recognition, characterized in that, The method comprises the following steps: A multi-modal perception module is used to collect multi-modal data and pre-process the multi-modal data to generate a standardized data stream; A cross-modal fusion and emotion recognition module is used to receive the standardized data stream and interactively model multi-modal features to output a current emotion label and an emotion intensity; The cross-modal fusion and emotion recognition module comprises a cross-modal graph neural network unit, a cross-modal contrast learning unit, a time sequence attention network unit, and a sequence smoothing unit; The cross-modal graph neural network unit receives each modality feature output by the multi-modal perception module, maps the modality features into independent sub-nodes in a graph structure respectively, and models the association between the independent sub-nodes by performing a preset graph convolution operation and calling a configured learnable edge weight matrix; The cross-modal contrast learning unit constructs a positive sample pair and a negative sample pair based on the modality features in the model training stage, and uses a preset contrast loss function to maximize the similarity of the modality features corresponding to the positive sample pair and minimize the similarity of the modality features corresponding to the negative sample pair; The time sequence attention network unit receives a multi-modal fusion feature sequence output by the cross-modal graph neural network unit, performs self-attention calculation on the multi-modal fusion feature sequence, and assigns different time dimension attention weights to the multi-modal fusion features at different time steps in the feature sequence through the self-attention mechanism; The sequence smoothing unit receives an emotion recognition label sequence output by the time sequence attention network unit, and uses a preset sequence smoothing model to constrain the correlation between the emotion recognition label sequences at adjacent time steps to output a current emotion label and an emotion intensity; A reaction planning module is used to generate a composite reaction strategy based on the current emotion label, the emotion intensity, and a historical emotional trajectory; A digital human rendering module is used to map the composite reaction strategy into control signals corresponding to speech, facial expressions, and actions respectively, and drive the digital human to perform corresponding speech output, facial expression changes, and limb actions through the control signals to realize interaction.

2. The digital human interaction system based on multi-modal emotion recognition according to claim 1, characterized in that, The pre-processing of the multi-modal data to generate a standardized data stream comprises: Performing denoising processing on the speech signal and extracting the Mel spectrum and prosody features; Performing semantic vectorization processing on the text based on a pre-trained language representation model; Extracting facial micro-expression and facial dynamic features from the visual signal through a three-dimensional convolution network; Extracting limb action features from the posture signal based on a graph convolution network; Extracting physiological features from the physiological signal through one-dimensional convolution and time sequence modeling; Aligning and standardizing the above-mentioned modality features in the time dimension to generate a standardized data stream.

3. The digital human interaction system based on multi-modal emotion recognition according to claim 1, characterized in that, The cross-modal graph neural network unit comprises: A modality confidence evaluation sub-unit that calculates the confidence score of each modality based on a short-term consistency indicator, a confidence output by a pre-trained encoder, and / or a signal-to-noise ratio; The dynamic edge weight updating subunit performs adaptive regularization on the learnable edge weight matrix to weaken the weight value of the edge corresponding to the modal feature in the graph structure when the confidence score is lower than a predetermined threshold, or when it is detected that the noise of the modal feature exceeds a predetermined noise threshold, or when it is detected that the condition of missing modal features is met.

4. The digital human interaction system based on multi-modal emotion recognition according to claim 3, characterized in that, The adaptive regularization of the learnable edge weight matrix includes obtaining actual edge weights through double-scale weight maintenance.

5. The digital human interaction system based on multi-modal emotion recognition according to claim 1, characterized in that, The generation of the composite response strategy according to the current emotional label, the emotional intensity and the historical emotional trajectory includes: The historical emotional trajectory of the user is modeled by using the emotional memory network to predict the transfer trend of the emotional state of the user. A candidate strategy generator is used to generate multiple candidate composite response strategies based on the predicted transfer trend of the emotional state of the user and the current emotional label by using a sequence-to-sequence neural network with an attention mechanism. A strategy planner is used to evaluate and select the candidate composite response strategies by using a reinforcement learning method based on a value function to obtain the composite response strategy.

6. The digital human interaction system based on multi-modal emotion recognition according to claim 5, characterized in that, The strategy planner uses an interaction effect index as a reward signal, and the interaction effect index includes the improvement degree of the emotional state of the user and the improvement value of the user's participation.

7. The digital human interaction system based on multi-modal sentiment recognition according to claim 1, characterized in that, The composite response strategy includes an emotional response intensity parameter, and the emotional response intensity parameter is in a positive correlation mapping relationship with the emotional intensity output by the cross-modal fusion and emotional recognition module.

8. The digital human interaction system based on multi-modal emotion recognition according to claim 1, characterized in that, The digital human rendering module includes: A speech synthesis unit decodes the composite response strategy by using a sequence-to-sequence neural network based on a Tacotron architecture to generate a voice control signal. A facial expression generation unit generates a facial expression control signal based on a deformation parameterization model of a BlendShape basic expression component and in combination with an attention weight associated with the current emotional label. An action generation unit constructs a graph structure with human skeleton joints as nodes, predicts the joint position trajectory of the next moment under the constraint of the composite response strategy by using a graph convolution network, and generates an action control signal. A timing synchronization unit registers the voice control signal, the facial expression control signal and the action control signal by using a dynamic time warping algorithm, and compensates for the unaligned part by using a timestamp interpolation method.

9. A digital human interaction method based on multi-modal emotion recognition, characterized in that, The digital human interaction method based on multi-modal emotional recognition includes: Collecting multi-modal data and preprocessing the multi-modal data to generate standardized data streams; Receiving the standardized data streams and modeling the multi-modal features to output the current emotional label and the emotional intensity; the cross-modal fusion and emotional recognition module includes a cross-modal graph neural network unit, a cross-modal contrast learning unit, a timing attention network unit and a sequence smoothing unit; The cross-modal graph neural network unit receives each modal feature output by the multi-modal perception module, maps the modal features into independent sub-nodes in a graph structure respectively, and models the association relationship between the independent sub-nodes by performing a preset graph convolution operation and calling a configured learnable edge weight matrix. The cross-modal contrast learning unit constructs positive sample pairs and negative sample pairs based on the features of each modality during the model training phase, and uses a preset contrast loss function to maximize the similarity of the modality features corresponding to the positive sample pairs while minimizing the similarity of the modality features corresponding to the negative sample pairs. The time sequence attention network unit receives the multi-modal fusion feature sequence output by the cross-modal graph neural network unit and performs self-attention calculation on the multi-modal fusion feature sequence. The self-attention mechanism assigns different time dimension attention weights to the multi-modal fusion features at different time steps in the feature sequence. The sequence smoothing unit receives the emotion recognition label sequence output by the time sequence attention network unit, and uses a preset sequence smoothing model to associate the emotion recognition label sequences at adjacent time steps and output the current emotion label and emotion intensity. According to the current emotion label, the emotion intensity, and the historical emotional trajectory, a composite reaction strategy is generated. The composite reaction strategy is mapped to control signals corresponding to voice, facial expressions, and actions, respectively, and the digital person is driven to perform corresponding voice output, facial expression changes, and limb movements to achieve interaction through the control signals.

Images