Multi-modal emotion recognition and interaction adjusting system and method based on uncertainty evaluation

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal emotion recognition and interaction adjusting system and method based on uncertainty evaluation, and solves the defects of uncertainty processing, robustness of interaction strategies, complementary information mining degree among modals and the like in the man-machine interaction process in the prior art. Depth application finiteness is caused by lack of modeling for feature uncertainty after fusion. The uncertainty of an emotion recognition result is quantified through technologies such as multi-modal feature fusion and Bayesian neural network / model integration, an interaction strategy is dynamically adjusted according to an uncertainty score, emotion clarification or conservative response is triggered in a high-uncertainty scene, the robustness of human-computer interaction and the user experience are improved, and the user experience is improved. The method is suitable for intelligent customer service, government affair consultation, medical inquiry and other scenes with high requirements for emotion interaction accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal emotion recognition and interaction adjustment system and method based on uncertainty assessment. Background Technology

[0002] "Multimodal emotion" refers to the study and understanding of emotions by integrating information from multiple different modalities. Modality here refers to the way information is presented or expressed, such as language, tone of voice, facial expressions, and body language. "Uncertainty assessment" refers to the process of analyzing, measuring, and judging the uncertainties inherent in an event, phenomenon, decision, or system.

[0003] In human-computer interaction scenarios such as intelligent question answering and intelligent customer service, emotion assessment and recognition are key technologies for achieving natural and human-like human-computer dialogue. However, existing technologies using multimodal fusion to improve the accuracy of emotion recognition generally face the following technical bottlenecks:

[0004] The lack of handling of uncertainty: Existing models directly output the sentiment category with the highest probability, without quantifying the uncertainty of the recognition results (such as the model's confidence in judging ambiguous inputs), which may lead to the system making incorrect interaction decisions based on low confidence results.

[0005] Insufficient robustness of interaction strategies: When users' emotional expression is ambiguous (such as micro-expressions or ambiguous tone) or there are individual differences, incorrect emotion recognition may lead to inappropriate responses (such as misjudging user confusion as neutral, resulting in insufficient interpretation) and reduce user experience.

[0006] Traditional feature fusion methods (such as simple cascading) do not fully exploit complementary information between modalities and lack modeling of uncertainties in the fused features. Therefore, the deep application of existing technologies in multimodal fusion is still relatively limited. In addition, although a few studies have attempted to assess uncertainty through Bayesian models or model ensembles, a complete closed loop of "identification-assessment-interaction adjustment" has not yet been formed. Especially in scenarios with high requirements for interaction accuracy, such as government affairs and healthcare, there is an urgent need for a technical solution that can quantify uncertainty and dynamically adapt interaction strategies.

[0007] Based on the above reasons, this invention designs a multimodal emotion recognition and interaction adjustment system and method based on uncertainty assessment. By integrating multimodal feature fusion, uncertainty assessment and interaction strategies are adapted to form a complete system architecture, which can accurately quantify the uncertainty of multimodal emotion recognition results and effectively improve the robustness of human-computer interaction and user experience. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal emotion recognition and interaction adjustment system and method based on uncertainty assessment. By integrating multimodal feature fusion, uncertainty assessment and interaction strategies are adapted to form a complete system architecture, which can accurately quantify the uncertainty of multimodal emotion recognition results and effectively improve the robustness of human-computer interaction and user experience.

[0009] To achieve the above objectives, the present invention provides a multimodal emotion recognition and interaction adjustment system based on uncertainty assessment, comprising a multimodal data processing unit, a feature fusion unit, an emotion recognition and uncertainty assessment unit, an interaction strategy adjustment unit, a response generation unit, a storage module, and a processor. The storage module is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, and the interaction strategy adjustment unit. The processor is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, and the response generation unit. The multimodal data processing unit is communicatively connected to the feature fusion unit, the feature fusion unit is communicatively connected to the emotion recognition and uncertainty assessment unit, the emotion recognition and uncertainty assessment unit is communicatively connected to the interaction strategy adjustment unit, and the interaction strategy adjustment unit is communicatively connected to the response generation unit.

[0010] The multimodal data processing unit includes a text preprocessing module, a speech preprocessing module, and a visual preprocessing module; the feature fusion unit includes a unimodal feature extraction subunit and a multimodal fusion subunit; the emotion recognition and uncertainty assessment unit includes an emotion classification model and an uncertainty quantification module; the interaction strategy adjustment unit includes a threshold comparison module and a strategy triggering module; the response generation unit includes a natural language generation module and a speech synthesis module; the storage module is used to store pre-trained model parameters, threshold configurations, interaction strategy templates, and knowledge base data; the processor executes the control logic of the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, the response generation unit, and the storage module, and supports data flow and collaboration between the modules.

[0011] The single-modal feature extraction subunit includes a text encoder, a speech network, and a visual network; the multimodal fusion subunit includes an attention mechanism or a fusion network; the uncertainty quantification module includes a BNN, a model ensemble, or an evidence theory module; and the policy triggering module includes an emotion matching subunit, a clarification subunit, and a conservative policy subunit.

[0012] A multimodal emotion recognition and interaction adjustment method based on uncertainty assessment includes the following steps:

[0013] S1, using a multimodal data processing unit to acquire and preprocess multimodal data, including:

[0014] S1-1, Data Acquisition: Acquire the user's multimodal data through microphone, camera, and text input box interfaces. The multimodal data includes text data, voice data, and visual data.

[0015] S1-2, Data Preprocessing:

[0016] S1-2-1 performs text processing through word segmentation, stop word removal, part-of-speech tagging, and generation of word vector sequences;

[0017] S1-2-2 performs speech processing by endpoint detection, noise reduction and framing, and extracting MFCC, pitch, speech rate and energy acoustic features.

[0018] S1-2-3 performs visual processing through face detection, key point localization, calculation of facial motion units and expression sub-descriptions;

[0019] S2, using a feature fusion unit to fuse multimodal features, including;

[0020] S2-1, Single-modal feature extraction, specifically includes:

[0021] S2-1-1, Text Features: Extracting contextual semantic feature vectors using BiLSTM or Transformer models.

[0022] Among them, the sentiment feature vector of the text modality is extracted by the text encoder, and dt represents the dimension of the text feature vector;

[0023] S2-1-2, Speech Features: Extracting Temporal Acoustic Feature Vectors using a CNN-LSTM Network

[0024] The emotion feature vector of the speech modality is extracted through a speech feature network (such as a CNN-LSTM network), d a It is the dimension of the speech feature vector.

[0025] S2-1-3, Visual Features: Extracting facial expression feature vectors using the ResNet+GCN model.

[0026] The emotion feature vector of the visual modality is extracted by a facial expression recognition network (such as the ResNet+GCN model), d v is the dimension of the visual feature vector.

[0027] S2-2, Multimodal Fusion: Weighted fusion using an attention mechanism, the specific formula is as follows:

[0028] Formula 1: α = Softmax(W1·[T; A; V]);

[0029] [T;A;V] concatenates the feature vectors of text, speech, and vision to form a new vector.

[0030] Formula 2: F = α T T+α A A+α V V;

[0031] α: The attention weight vector obtained after calculation by the Softmax function, whose elements α T α A α V The weights correspond to the text, speech, and visual modalities, respectively, and satisfy α. T +α A +α V =1 indicates the relative importance of each mode in the fusion process;

[0032] Where, α T α A and α V For the attention weights of each modality, Features of fusion;

[0033] S3 uses an emotion recognition and uncertainty assessment unit to perform emotion recognition and uncertainty quantification, including:

[0034] S3-1, Model Input: Input the fused features into the sentiment classification model for sentiment recognition;

[0035] S3-2, Model Output: An uncertainty score is given based on the emotion recognition results, specifically including:

[0036] S3-2-1, Sentiment prediction results, including discrete categories or continuous coordinates;

[0037] S3-2-2, Uncertainty Score (U-score): Specific calculation methods include Bayesian Neural Networks (BNNs), model ensembles, evidence theory, or a combination thereof.

[0038] Bayesian Neural Network (BNN): Models the weight parameter w as a distribution By sampling N times using MC, we obtain the third formula for the predicted distribution:

[0039] w: Weight parameters in a Bayesian neural network. In traditional neural networks, weights are fixed values, while in BNNs, weights are modeled as probability distributions.

[0040] F: This refers to the multimodal fusion feature vector obtained from the previous multimodal feature fusion step, which serves as the input to the BNN.

[0041] p(y|F,w): Given the input features F and weight parameters w, the model's predicted probability for the sentiment category y.

[0042] p(y|F,D): The final prediction distribution obtained by integrating the posterior distribution of the weight parameter w (approximated by MC sampling in practice) represents the probability of the predicted sentiment category after considering the uncertainty of the model parameters.

[0043] y: Emotion category variable, with values ​​ranging from a predefined set of emotion categories (such as discrete categories like happiness, sadness, and anger, or continuous coordinates in the valence-arousal space).

[0044] Calculate the prediction entropy, i.e., Formula 4: As an uncertainty score, the U-score;

[0045] H: Prediction entropy, used to quantify the prediction distribution. The greater the entropy value, the higher the uncertainty of the model regarding the prediction result. This value is used as the uncertainty score, U-score.

[0046] Model ensemble: Training M independent models Calculate the variance of the prediction results:

[0047] That is, Formula 5: in To integrate the average predicted values;

[0048] M: The number of ensemble models, i.e., the number of emotion recognition models trained with the same or different structures.

[0049] yi: The sentiment prediction result of the i-th ensemble model on the input data, i = 1, 2, ..., M. The prediction result can be a discrete sentiment category, or a continuous sentiment value or category probability.

[0050] The average of the predictions from M ensemble models.

[0051] σ 2 The variance of the prediction results measures the degree of difference between the prediction results of the M ensemble models. The larger the variance, the greater the prediction discrepancy between the models, that is, the higher the uncertainty in the sentiment recognition of the current input.

[0052] Evidence theory: Output the evidence e for each category c. c To calculate the uncertainty mass, use Formula Six:

[0053] c: represents different values ​​for the sentiment category;

[0054] e c: Indicates the amount of evidence for a specific sentiment category c;

[0055] ∑ c e c : is the amount of evidence e for all sentiment categories c c Perform a summation operation;

[0056] u: This refers to the uncertainty quality, which is the final result calculated by the formula and is used to quantify the degree of uncertainty of the emotion recognition result. The value of u ranges from 0 to 1. The larger the value, the higher the uncertainty of the model regarding the current emotion recognition result; conversely, the smaller the value of u, the more confident the model is in the recognition result and the lower the uncertainty.

[0057] S4, using the interaction strategy adjustment unit to dynamically adjust the interaction strategy, including:

[0058] S4-1, Threshold Setting: Pre-screening Low Uncertainty Threshold U low and high uncertainty threshold U high The threshold is optimized using the validation set;

[0059] S4-2, the strategy triggering logic, is as follows:

[0060] S4-2-1, if U-score≤U low The sentiment matching strategy is executed through the sentiment matching submodule, and a corresponding response is generated based on the sentiment prediction results of S3-2-1.

[0061] S4-2-2, if U-score≥U high This triggers the emotional clarification mechanism of the clarification submodule or the conservative strategy of the conservative strategy submodule.

[0062] The clarification mechanism is as follows: output preset clarification statements or provide an interface for selecting emotional tags;

[0063] The conservative strategy is to respond in a neutral tone, avoid emotional expressions, and increase interactive options;

[0064] S4-2-3, if there is moderate uncertainty, i.e., U low <U-score≤U high Implement a compromise strategy, that is, within the standard response...

[0065] Add tentative statements;

[0066] S5, combining the adjusted interaction strategy from S4 with the user's question intent, uses the response generation unit to generate and output the response.

[0067] Output includes both text and voice responses, specifically:

[0068] Text response: generated through template matching or seq2seq models, with wording adjusted based on sentiment tone;

[0069] Voice response: Synthesized using a TTS engine, the speech rate and tone are adjusted according to the emotion being expressed.

[0070] The text data in S1-1 includes chat input and historical conversations; the voice data includes real-time voice streams with a sampling rate of ≥16kHz; and the visual data includes facial images or video streams with a resolution of ≥640×480 for key areas of the eyes and mouth.

[0071] The threshold validation set optimization in S4-1 maximizes interaction satisfaction; when the sentiment prediction result in S4-2-1 is identified as confused, the response is to call the detailed explanation document in the knowledge base.

[0072] The steps of the emotion clarification mechanism are as follows:

[0073] Generate clarifying questions: Select the corresponding template based on the predicted sentiment category;

[0074] Receive user feedback: Supports text input, emotion tag clicks, or voice confirmation;

[0075] Feedback processing: If the user selects a specific emotion tag, update the current emotion status; if the user does not provide feedback, proceed with the conservative strategy process.

[0076] The uncertainty quantization training steps for a Bayesian neural network (BNN) are as follows:

[0077] Data preparation: Collect a multimodal sentiment dataset containing text, speech, and images, and label discrete sentiment categories and continuous valence-arousal values;

[0078] Network structure: Variational dropout is introduced into the output layers of the text encoder Transformer, the speech network CNN-LSTM, and the visual network ResNet to transform the deterministic network into a BNN;

[0079] Loss function: Jointly optimizes the sentiment classification loss and uncertainty regularization term;

[0080] Inference phase: The entropy value of the predicted distribution is calculated through N=20 MC samplings and used as the U-score.

[0081] S4-1 specifically involves using a grid search method to determine the threshold U. low and U high This includes the following steps:

[0082] Define evaluation metrics: user subjective rating of interaction satisfaction, clarification request acceptance rate, and error sentiment response rate;

[0083] Iterate through the threshold combinations and calculate the weighted score for each indicator;

[0084] Choose the threshold combination that yields the highest weighted score as the final configuration.

[0085] Threshold combination is U low ∈[0.2,0.4],U high ∈[0.6,0.8]).

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] Uncertainty Quantification Mechanism: This invention quantifies the confidence level of emotion recognition through technologies such as BNN and model ensemble, solving the problem of "overconfidence" in traditional models and enabling the system to recognize the reliability of the results.

[0088] Robust Interaction Strategy: This invention designs an emotional clarification and conservative response strategy mechanism for high uncertainty scenarios to avoid interaction errors caused by erroneous emotional judgments, and significantly reduces communication risks, especially in sensitive scenarios such as government consultation and medical consultation.

[0089] Multimodal deep fusion: This invention uses a dynamic weighted fusion method based on attention mechanism to effectively capture complementary information between text, speech and visual modalities, improving the accuracy of emotion recognition by 15%-20% (compared to traditional cascaded fusion).

[0090]

[0091] Personalized Interaction Adaptation: This invention supports flexible configuration for different scenarios (such as customer service and education) through dynamic threshold adjustment and strategy grading, thus meeting diverse interaction needs. Attached Figure Description

[0092] Figure 1 This is a flowchart illustrating the multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to the present invention.

[0093] Figure 2 This is a functional block diagram of the multimodal emotion recognition and interaction adjustment system based on uncertainty assessment according to the present invention.

[0094] Figure 3 This is a schematic diagram of the evaluation process of the uncertainty quantification module based on Bayesian neural network (BNN) of the present invention.

[0095] Figure 4 This is a schematic diagram of the uncertainty quantification module evaluation process based on module integration of the present invention.

[0096] Figure 5 This is an interactive logic diagram of the emotion clarification mechanism process of the present invention. Detailed Implementation

[0097] The present invention will now be further described with reference to the accompanying drawings.

[0098] See Figures 1-5 The present invention provides a multimodal data processing unit, a feature fusion unit, an emotion recognition and uncertainty assessment unit, an interaction strategy adjustment unit, a response generation unit, a storage module, and a processor. The storage module is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, and the interaction strategy adjustment unit. The processor is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, and the response generation unit. The multimodal data processing unit is communicatively connected to the feature fusion unit, the feature fusion unit is communicatively connected to the emotion recognition and uncertainty assessment unit, the emotion recognition and uncertainty assessment unit is communicatively connected to the interaction strategy adjustment unit, and the interaction strategy adjustment unit is communicatively connected to the response generation unit.

[0099] The multimodal data processing unit includes a text preprocessing module, a speech preprocessing module, and a visual preprocessing module; the feature fusion unit includes a unimodal feature extraction subunit and a multimodal fusion subunit; the emotion recognition and uncertainty assessment unit includes an emotion classification model and an uncertainty quantification module; the interaction strategy adjustment unit includes a threshold comparison module and a strategy triggering module; the response generation unit includes a natural language generation module and a speech synthesis module; the storage module is used to store pre-trained model parameters, threshold configurations, interaction strategy templates, and knowledge base data; the processor executes the control logic of the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, the response generation unit, and the storage module, and supports data flow and collaboration between the modules.

[0100] The single-modal feature extraction subunit includes a text encoder, a speech network, and a visual network; the multimodal fusion subunit includes an attention mechanism or a fusion network; the uncertainty quantification module includes a BNN, a model ensemble, or an evidence theory module; and the policy triggering module includes an emotion matching subunit, a clarification subunit, and a conservative policy subunit.

[0101] A multimodal emotion recognition and interaction adjustment method based on uncertainty assessment includes the following steps:

[0102] S1, using a multimodal data processing unit to acquire and preprocess multimodal data, including:

[0103] S1-1, Data Acquisition: Acquire the user's multimodal data through microphone, camera, and text input box interfaces. The multimodal data includes text data, voice data, and visual data.

[0104] S1-2, Data Preprocessing:

[0105] S1-2-1, text processing is performed by generating word vector sequences through jieba word segmentation, stop word removal, part-of-speech tagging, Word2Vec or BERT embedding;

[0106] S1-2-2 uses VAD algorithm endpoint detection, spectral subtraction for noise reduction, and 25ms frame length with 10ms frame shift to extract 13-dimensional MFCC, pitch, speech rate and energy acoustic features for speech processing.

[0107] MFCC (Mel Frequency Cepstral Coefficients) is a feature parameter widely used in speech signal processing and speech recognition. In this patent, it is used for feature extraction from speech data. The following section introduces its principles, calculation process, and application advantages:

[0108] MFCC (Multi-Frequency Coherence to the Mel Scale) is based on the characteristics of the human auditory system. It transforms speech signals from the time domain to the frequency domain and then performs a nonlinear transformation using the Mel Scale to obtain characteristic parameters. The human auditory system's perception of sounds at different frequencies is not linear. The Mel Scale simulates this nonlinear perception characteristic, mapping linear frequencies to Mel frequencies. This makes the characteristics in the Mel frequency domain more consistent with the laws of human auditory perception and better reflects the essential characteristics of the speech signal.

[0109] The calculation process of MFCC includes:

[0110] Pre-emphasis: High-pass filtering is applied to the speech signal to enhance the energy of the high-frequency components, compensate for the high-frequency attenuation of the speech signal caused by lip radiation, and enhance the high-frequency details in the speech.

[0111] Framing and windowing: The speech signal is divided into short frames (usually 20-30ms long, with a frame shift of 10ms), and each frame is windowed (such as a Hamming window) to reduce the spectral leakage caused by signal truncation.

[0112] Fast Fourier Transform (FFT): Converts each frame of time-domain signal into a frequency-domain signal to obtain the spectrum of the speech signal.

[0113] Mel filter bank: A set of bandpass filters with equal intervals on the Mel scale are used to filter the spectrum, converting the linear frequency axis to the Mel frequency axis, and calculating the energy of the output of each filter.

[0114] Logarithmic operation and Discrete Cosine Transform (DCT): Take the logarithm of the energy output of each filter to compress the dynamic range, and then perform a discrete cosine transform to obtain the MFCC coefficients. Usually, the first 12-13 coefficients are taken as speech features because these coefficients contain the main information of the speech signal.

[0115] This patent employs MFCC to extract speech features, effectively capturing acoustic information related to emotional expression in speech signals, such as the differences in the spectral envelope of speech under different emotional states. Compared to other speech feature extraction methods, MFCC has a better ability to represent information such as timbre, pitch, and formants in speech, and is robust to noise, providing stable and effective speech modal features for subsequent multimodal feature fusion and emotion recognition.

[0116] S1-2-3, face detection using the MTCNN algorithm, localization of 68 facial landmarks, and AUS, and calculation of facial action units and expression sub-descriptions (such as the degree of mouth corner raising and eyebrow curvature) based on FACS encoding for visual processing; S2, multimodal feature fusion using a feature fusion unit, including;

[0117] S2-1, Single-modal feature extraction, specifically includes:

[0118] S2-1-1, Text Features: Extracting contextual semantic feature vectors using BiLSTM or Transformer models.

[0119] Among them, the sentiment feature vector of the text modality is extracted by the text encoder, and dt represents the dimension of the text feature vector;

[0120] S2-1-2, Speech Features: Extracting Temporal Acoustic Feature Vectors using a CNN-LSTM Network

[0121] The emotion feature vector of the speech modality is extracted through a speech feature network (such as a CNN-LSTM network), d a It is the dimension of the speech feature vector.

[0122] S2-1-3, Visual Features: Extracting facial expression feature vectors using the ResNet+GCN model.

[0123] The emotion feature vector of the visual modality is extracted by a facial expression recognition network (such as the ResNet+GCN model), d v is the dimension of the visual feature vector.

[0124] S2-2, Multimodal Fusion: Weighted fusion using an attention mechanism, the specific formula is as follows:

[0125] Formula 1: α = Softmax(W1·[T; A; V]);

[0126] [T;A;V] concatenates the feature vectors of text, speech, and vision to form a new vector.

[0127] Formula 2: F = α T T+α A A+α V V;

[0128] α: The attention weight vector obtained after calculation by the Softmax function, whose elements α T α A α V The weights correspond to the text, speech, and visual modalities, respectively, and satisfy α. T +α A +α V =1 indicates the relative importance of each mode in the fusion process.

[0129] Where, α T α A and α V For the attention weights of each modality, Features of fusion;

[0130] S3 uses an emotion recognition and uncertainty assessment unit to perform emotion recognition and uncertainty quantification, including:

[0131] S3-1, Model Input: Input the fused features into the sentiment classification model for sentiment recognition;

[0132] S3-2, Model Output: An uncertainty score is given based on the emotion recognition results, specifically including:

[0133] S3-2-1, Emotion prediction results, including discrete categories (such as {happy, sad, angry, confused, neutral}) or continuous coordinates (valence-arousal space, range [-1,1]×[-1,1]);

[0134] S3-2-2, Uncertainty Score (U-score): Specific calculation methods include Bayesian Neural Networks (BNNs), model ensembles, evidence theory, or a combination thereof.

[0135] Bayesian Neural Network (BNN): Models the weight parameter w as a distribution By sampling N times using MC, we obtain the third formula for the predicted distribution:

[0136] w: Weight parameters in a Bayesian neural network. In traditional neural networks, weights are fixed values, while in BNNs, weights are modeled as probability distributions.

[0137] F: This refers to the multimodal fusion feature vector obtained from the previous multimodal feature fusion step, which serves as the input to the BNN.

[0138] p(y|F,w): Given the input features F and weight parameters w, the model's predicted probability for the sentiment category y.

[0139] The final prediction distribution obtained by integrating the posterior distribution of the weight parameter w (approximated in practice by MC sampling) represents the probability of the predicted sentiment category after considering the uncertainty of the model parameters.

[0140] y: Emotion category variable, with values ​​ranging from a predefined set of emotion categories (such as discrete categories like happiness, sadness, and anger, or continuous coordinates in the valence-arousal space).

[0141] Calculate the prediction entropy, i.e., Formula 4: As an uncertainty score, the U-score;

[0142] H: Prediction entropy, used to quantify the prediction distribution. The greater the entropy value, the higher the uncertainty of the model regarding the prediction result. This value is used as the uncertainty score, U-score.

[0143] Model ensemble: Training M independent models Calculate the variance of the prediction results:

[0144] That is, Formula 5: in To integrate the average predicted values;

[0145] M: The number of ensemble models, i.e., the number of emotion recognition models trained with the same or different structures.

[0146] yi: The sentiment prediction result of the i-th ensemble model on the input data, i = 1, 2, ..., M. The prediction result can be a discrete sentiment category, or a continuous sentiment value or category probability.

[0147] The average of the predictions from M ensemble models.

[0148] σ 2 The variance of the prediction results measures the degree of difference between the prediction results of the M ensemble models. The larger the variance, the greater the prediction discrepancy between the models, that is, the higher the uncertainty in the sentiment recognition of the current input.

[0149] Evidence theory: Output the evidence e for each category c. c To calculate the uncertainty mass, use Formula Six:

[0150] c: Represents different values ​​for the emotion category. In emotion recognition tasks, the emotion category is a predefined series of possible emotion states. For example, in common emotion classification systems, c may correspond to discrete emotion categories such as "happy," "sad," "angry," "confused," and "neutral"; it can also be different emotion regions in continuous emotion representations such as valence-arousal space. By analyzing and calculating each c, the uncertainty of the emotion recognition results can be comprehensively evaluated.

[0151] e c This represents the amount of evidence for a specific sentiment category c. Here, "evidence" refers to the relevant grounds or information measures that support a particular sentiment category c when the model is performing sentiment recognition. Its specific calculation method depends on the evidence theory model and related algorithms used. For example, it may be derived from a combination of factors such as the model's probability of predicting that category, the degree of feature matching, and the support in logical reasoning, used to quantify the strength of the model's evidence that the current input belongs to category c.

[0152] ∑ c e c : is the amount of evidence e for all sentiment categories c c Perform a summation operation. The summation result reflects the overall accumulation of evidence from all possible sentiment categories when the model performs sentiment recognition on the current input. By summing the evidence from all categories, a comprehensive total amount of evidence can be obtained, which is used for subsequent calculation of uncertainty quality.

[0153] u: This refers to the uncertainty mass, which is the final result calculated by the formula and is used to quantify the degree of uncertainty in the emotion recognition result. The value of u ranges from 0 to 1. A larger value indicates higher uncertainty about the current emotion recognition result; conversely, a smaller u value indicates greater confidence in the recognition result and lower uncertainty. Calculating the uncertainty mass u provides important reference for adjusting subsequent interaction strategies during the emotion recognition process. For example, when u exceeds a certain threshold, emotion clarification or conservative interaction strategies can be triggered to cope with high uncertainty.

[0154] S4, using the interaction strategy adjustment unit to dynamically adjust the interaction strategy, including:

[0155] S4-1, Threshold Setting: Pre-screening Low Uncertainty Threshold U low and high uncertainty threshold U high The threshold is optimized using the validation set;

[0156] S4-2, the strategy triggering logic, is as follows:

[0157] S4-2-1, if U-score≤U lowThe sentiment matching submodule executes the sentiment matching strategy and generates a corresponding response based on the sentiment prediction result of S3-2-1 (e.g., when it is identified as "confused", it calls the detailed explanation document in the knowledge base).

[0158] S4-2-2, if U-score≥U high This triggers the emotional clarification mechanism of the clarification submodule or the conservative strategy of the conservative strategy submodule.

[0159] The clarification mechanism is as follows: output preset clarification statements or provide an interface for selecting emotional tags;

[0160] A conservative strategy is to respond in a neutral tone, avoid emotional expressions (such as using "We understand your needs, here is some relevant information..." instead of "I'm happy to help you...") and increase interactive options (such as "Please click here for further explanation").

[0161] S4-2-3, if there is moderate uncertainty, i.e., U low <U-score≤U high Alternatively, a compromise strategy can be adopted, which involves adding probing questions (such as "Based on your expression, I assume you may be confused; would you like more details?") to the standard response.

[0162] S5, combining the adjusted interaction strategy from S4 with the user's question intent, uses the response generation unit to generate and output responses, including text and voice responses, specifically:

[0163] Text response: generated through template matching or seq2seq model, and wording adjusted according to sentiment tone (e.g., using exclamation marks to enhance sentiment in low uncertainty, and using concise declarative sentences in high uncertainty);

[0164] The seq2seq (Sequence to Sequence) model is a deep learning model architecture widely used in fields such as natural language processing. It is primarily used to solve tasks where both input and output are sequential data, and in the aforementioned patent, it is applied to the text response generation stage. The following section introduces its definition, structure, working principle, application scenarios, and role in the patent:

[0165] Definition: The seq2seq model is an end-to-end neural network architecture that maps one input sequence to another output sequence without requiring predefined fixed input and output lengths, and can handle sequence data of variable length. It overcomes the limitations of traditional neural networks with fixed input and output dimensions, playing a crucial role in tasks such as machine translation, text summarization, and question answering systems.

[0166] Structure: A seq2seq model typically consists of two parts: an encoder and a decoder.

[0167] The encoder is responsible for transforming an input sequence (such as a piece of text or a sentence) into a fixed-length vector representation, also known as a context vector, which contains the semantic information of the input sequence. Encoders typically employ recurrent neural networks (RNNs) and their variants, such as long short-term memory networks (LSTM) and gated recurrent units (GRUs), to progressively encode information into the context vector by processing the input sequence sequentially.

[0168] Decoder: Utilizing the context vector obtained from the encoder, the decoder progressively generates the target output sequence. Also based on RNN / LSTM / GRU, the decoder extracts information from the context vector and generates an output element at each time step until a complete output sequence (e.g., a translated sentence, or an answer to a question) is generated.

[0169] How it works: During training, the model adjusts the encoder and decoder parameters using backpropagation based on the input sequence and the corresponding target output sequence, minimizing the loss between the predicted output and the true output (such as cross-entropy loss). During inference, the model receives the input sequence, the encoder generates a context vector, and the decoder generates the output sequence word by word based on this vector. To improve generation quality, an attention mechanism is often introduced, allowing the decoder to dynamically focus on different parts of the input sequence when generating each element, rather than relying solely on a fixed context vector.

[0170] In the text response generation stage of this patent, the seq2seq model encodes and transforms the input semantic information into appropriate text responses based on the adjusted interaction strategy and the user's question intent. By learning from a large amount of dialogue data, it can generate response content that conforms to emotional tone and is semantically reasonable. For example, in low-uncertainty scenarios, it generates natural responses with emotional color; in high-uncertainty scenarios, it generates concise and neutral expressions, thereby achieving effective communication with users.

[0171] Voice response: Synthesized by a TTS engine, the speech rate and tone are adjusted according to the emotion matching strategy (e.g., "happy" emotion uses fast and high tone, "conservative strategy" uses medium speed and flat tone).

[0172] The text data in S1-1 includes chat input and historical conversations; the voice data includes real-time voice streams with a sampling rate of ≥16kHz; and the visual data includes facial images or video streams with a resolution of ≥640×480 for key areas of the eyes and mouth.

[0173] The threshold validation set optimization in S4-1 maximizes interaction satisfaction; when the sentiment prediction result in S4-2-1 is identified as confused, the response is to call the detailed explanation document in the knowledge base.

[0174] The steps of the emotion clarification mechanism are as follows:

[0175] Generate clarifying questions: Select the corresponding template based on the predicted sentiment category (e.g., if the prediction is "confused", use "Some information in your question was not clear enough. Could you please elaborate on your specific needs?").

[0176] Receive user feedback: Supports text input, emotion tag clicks (such as buttons "angry", "confused", "neutral"), or voice confirmation;

[0177] Feedback processing: If the user selects a specific emotion tag, update the current emotion status; if the user does not provide feedback, proceed with the conservative strategy process.

[0178] The uncertainty quantization training steps for a Bayesian neural network (BNN) are as follows:

[0179] Data preparation: Collect multimodal sentiment datasets (such as IEMOCAP, CMU-MOSEI) containing text, speech, and images, and label discrete sentiment categories and continuous valence-arousal values;

[0180] Network structure: Variational dropout is introduced into the output layers of the text encoder Transformer, the speech network CNN-LSTM, and the visual network ResNet to transform the deterministic network into a BNN;

[0181] Variational dropout is a technique used in deep learning for Bayesian neural networks (BNNs). It combines the ideas of traditional dropout and variational inference, aiming to model and quantify the uncertainty in neural networks, and plays an important role in the uncertainty quantification process of patents.

[0182] Definition: Variational dropout is an improvement on the traditional dropout method. Traditional dropout randomly sets the output of neurons to 0 with a fixed probability during training to prevent overfitting. Variational dropout, however, models the random process of dropout as a probability distribution, treating the dropout rate as a learnable parameter. It approximates the posterior distribution through variational inference, enabling the neural network to estimate the uncertainty of the prediction outcome. Instead of simply dropping neurons randomly during training, it models the probability of dropping each neuron, allowing the model to learn a more reasonable parameter distribution and thus quantify the model's cognitive uncertainty (the model's own uncertainty about the prediction outcome).

[0183] Working principle:

[0184] Probabilistic modeling: In variational dropout, for each layer of the neural network, an independent probability distribution is assigned to the connection of each neuron to describe the likelihood that the connection will be dropped during training. For example, for the weight matrix \(\mathbf{W}\), traditional dropout sets some elements to 0 with a fixed probability p; while variational dropout assumes that each weight element \(w_{ij}\) follows a certain distribution (such as a Bernoulli distribution), and the parameters of this distribution (such as the probability \(p_{ij}\) of the Bernoulli distribution) are learnable.

[0185] Variational inference: Since directly calculating the posterior distribution is often infeasible, variational dropout uses variational inference to introduce an approximate posterior distribution (such as a Gaussian distribution) to approximate the true posterior distribution. By minimizing the Kullback-Leibler divergence between the approximate and true posterior distributions, the parameters of the approximate posterior distribution are continuously adjusted, resulting in a more accurate parameter distribution estimate. During training, the network not only learns how to make predictions but also learns about parameter uncertainty, enabling the model to better handle noise and uncertainty in the data.

[0186] Applications in the prediction phase: During the inference phase, by sampling the probability distribution (such as Monte Carlo sampling), multiple different network structures can be obtained (because each sampling is equivalent to a different combination of discarded neurons), thus yielding multiple prediction results. The distribution of these prediction results can be used to quantify the uncertainty of the model, for example, by calculating the entropy, variance, and other metrics of the prediction results to measure the model's confidence in the current prediction.

[0187] In the uncertainty quantification process based on Bayesian neural networks in this patent, variational dropout is introduced into the output layers of the text encoder, speech network, and vision network, transforming the deterministic network into a Bayesian neural network. This approach enables the model to model uncertainty in the processing of multimodal features, thereby accurately quantifying the uncertainty score (U-score) while outputting sentiment prediction results, providing a reliable basis for subsequent adjustments to uncertainty-based interaction strategies.

[0188] Loss function: Jointly optimizes the sentiment classification loss (cross-entropy) and the uncertainty regularization term (KL divergence constraint weight distribution close to the prior);

[0189] Inference phase: The entropy value of the predicted distribution is calculated through N=20 MC samplings and used as the U-score.

[0190] S4-1 specifically involves using a grid search method to determine the threshold U. low and U high This includes the following steps:

[0191] Define evaluation metrics: user subjective rating of interaction satisfaction, clarification request acceptance rate, and error sentiment response rate; iterate through threshold combinations and calculate the weighted score of each metric;

[0192] Choose the threshold combination that yields the highest weighted score as the final configuration.

[0193] Threshold combination is U low ∈[0.2,0.4],U high ∈[0.6,0.8]).

[0194] The above are merely preferred embodiments of the present invention, intended only to aid in understanding the method and core ideas of this application. The scope of protection of the present invention is not limited to the above embodiments; all technical solutions falling within the scope of the present invention's concept are within its protection. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0195] This invention comprehensively addresses the shortcomings of existing technologies in human-computer interaction, such as uncertainty handling, robustness of interaction strategies, and the degree of complementary information mining between modalities, as well as the limitation in deep application due to the lack of modeling for the uncertainty of fused features. By quantifying the uncertainty of emotion recognition results through multimodal feature fusion and Bayesian neural networks / model ensemble techniques, and dynamically adjusting the interaction strategy based on the uncertainty score, it triggers emotion clarification or conservative responses in high-uncertainty scenarios, improving the robustness of human-computer interaction and user experience. It is suitable for scenarios with high requirements for the accuracy of emotional interaction, such as intelligent customer service, government consultation, and medical diagnosis.

Claims

1. A multimodal emotion recognition and interaction adjustment system based on uncertainty assessment, characterized in that, The system includes a multimodal data processing unit, a feature fusion unit, an emotion recognition and uncertainty assessment unit, an interaction strategy adjustment unit, a response generation unit, a storage module, and a processor. The storage module is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, and the interaction strategy adjustment unit. The processor is communicatively connected to the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, and the response generation unit. The multimodal data processing unit is communicatively connected to the feature fusion unit, the feature fusion unit is communicatively connected to the emotion recognition and uncertainty assessment unit, the emotion recognition and uncertainty assessment unit is communicatively connected to the interaction strategy adjustment unit, and the interaction strategy adjustment unit is communicatively connected to the response generation unit.

2. The multimodal emotion recognition and interaction adjustment system based on uncertainty assessment according to claim 1, characterized in that, The multimodal data processing unit includes a text preprocessing module, a speech preprocessing module, and a visual preprocessing module; the feature fusion unit includes a unimodal feature extraction subunit and a multimodal fusion subunit; the emotion recognition and uncertainty assessment unit includes an emotion classification model and an uncertainty quantification module; the interaction strategy adjustment unit includes a threshold comparison module and a strategy triggering module; the response generation unit includes a natural language generation module and a speech synthesis module; the storage module is used to store pre-trained model parameters, threshold configurations, interaction strategy templates, and knowledge base data; the processor executes the control logic of the multimodal data processing unit, the feature fusion unit, the emotion recognition and uncertainty assessment unit, the interaction strategy adjustment unit, the response generation unit, and the storage module, and supports data flow and collaboration among the modules.

3. The multimodal emotion recognition and interaction adjustment system based on uncertainty assessment according to claim 2, characterized in that, The single-modal feature extraction subunit includes a text encoder, a speech network, and a visual network; the multimodal fusion subunit includes an attention mechanism or a fusion network; the uncertainty quantification module includes a BNN, a model ensemble, or an evidence theory module; and the policy triggering module includes an emotion matching submodule, a clarification submodule, and a conservative policy submodule.

4. A method for multimodal emotion recognition and interaction adjustment based on uncertainty assessment, characterized in that, Includes the following steps: S1, using the multimodal data processing unit to acquire and preprocess multimodal data, including: S1-1, Data Acquisition: Acquire the user's multimodal data through the microphone, camera, and text input box interfaces. The multimodal data includes text data, voice data, and visual data. S1-2, Data Preprocessing: S1-2-1 performs text processing through word segmentation, stop word removal, part-of-speech tagging, and generation of word vector sequences; S1-2-2 performs speech processing by endpoint detection, noise reduction and framing, and extracting MFCC, pitch, speech rate and energy acoustic features. S1-2-3 performs visual processing through face detection, key point localization, calculation of facial motion units and expression sub-descriptions; S2, using the feature fusion unit to fuse multimodal features, including; S2-1, Single-modal feature extraction, specifically includes: S2-1-1, Text Features: Extracting contextual semantic feature vectors using BiLSTM or Transformer models. Among them, the sentiment feature vector of the text modality is extracted by the text encoder, and dt represents the dimension of the text feature vector; S2-1-2, Speech Features: Extracting Temporal Acoustic Feature Vectors using a CNN-LSTM Network The emotion feature vector of the speech modality is extracted through a speech feature network, d a This refers to the dimension of the speech feature vector; S2-1-3, visual features: extracting facial expression feature vectors using the ResNet+GCN model. The emotion feature vector of the visual modality is extracted by a facial expression recognition network, d v is the dimension of the visual feature vector. S2-2, Multimodal Fusion: Weighted fusion using an attention mechanism, the specific formula is as follows: Formula 1: α = Softmax(W1·[T; A; V]); [T; A; V] concatenates the feature vectors from text, speech, and vision modalities to form a new vector; Formula 2: F = α T T+α A A+α V V; α: The attention weight vector obtained after calculation by the Softmax function, whose elements α T α A α V The weights correspond to the text, speech, and visual modalities, respectively, and satisfy α. T +α A +α V =1 indicates the relative importance of each mode in the fusion process; Wherein, the α T α A and α V For the attention weights of each modality, Features of fusion; S3, using the aforementioned emotion recognition and uncertainty assessment unit to perform emotion recognition and uncertainty quantification, including: S3-1, Model Input: Input the fused features into the emotion classification model for emotion recognition; S3-2, Model Output: An uncertainty score is calculated based on the emotion recognition results, specifically including: S3-2-1, Sentiment prediction results, which include discrete categories or continuous coordinates; S3-2-2, Uncertainty Score (U-score): Specific calculation methods include Bayesian Neural Networks (BNNs), model ensembles, evidence theory, or a combination thereof. The Bayesian neural network (BNN) models the weight parameter w as a distribution. By sampling N times using MC, we obtain the third formula for the predicted distribution: w: Weight parameters in a Bayesian neural network, where the weights are modeled as a probability distribution; F: The multimodal fusion feature vector obtained in the multimodal feature fusion step of S2-2, which is used as the input of the BNN. p(y|F,w): Given the input features F and weight parameters w, the probability of the model predicting the sentiment category y; The final prediction distribution obtained by integrating the posterior distribution of the weight parameter w is the probability of the predicted sentiment category after considering the uncertainty of the model parameters. y: A sentiment category variable, whose value range is a predefined set of sentiment categories; Calculate the prediction entropy, i.e., Formula 4: The uncertainty score U-score; H: prediction entropy, used to quantify the prediction distribution. The greater the entropy value, the higher the uncertainty of the model regarding the prediction result. This value is used as the uncertainty score, U-score. The model ensemble consists of training M independent models. Calculate the variance of the prediction results: i.e., Formula 5: in To integrate the average predicted values; M: The number of ensemble models, i.e., the number of emotion recognition models trained with the same or different structures; yi: The sentiment prediction result of the i-th ensemble model on the input data, i = 1, 2, ..., M; the prediction result is either a discrete sentiment category or a continuous sentiment value or category probability; The average of the predictions from M ensemble models; σ 2 The variance of the prediction results measures the degree of difference between the prediction results of the M ensemble models. The larger the variance, the greater the prediction discrepancy between the models, that is, the higher the uncertainty in the sentiment recognition of the current input. The evidence theory states that it outputs evidence e for each category c. c To calculate the uncertainty mass, use Formula Six: c: represents different values ​​for the sentiment category; e c : Indicates the amount of evidence for a specific sentiment category c; ∑ c e c : is the amount of evidence e for all sentiment categories c c Perform a summation operation; u: This refers to the uncertainty quality, which is the final result calculated by the formula and is used to quantify the degree of uncertainty of the emotion recognition result. The value of u ranges from 0 to 1. The larger the value, the higher the uncertainty of the model regarding the current emotion recognition result; conversely, the smaller the value of u, the more confident the model is in the recognition result and the lower the uncertainty. S4, dynamically adjust the interaction strategy using the interaction strategy adjustment unit, including: S4-1, Threshold Setting: Pre-screening Low Uncertainty Threshold U low and high uncertainty threshold U high The threshold is optimized using a validation set; S4-2, the strategy triggering logic, is as follows: S4-2-1, if the U-score ≤ U low The emotion matching submodule executes the emotion matching strategy and generates a corresponding response based on the emotion prediction result of S3-2-1. S4-2-2, if the U-score ≥ U high This triggers the emotional clarification mechanism of the clarification submodule or the conservative strategy of the conservative strategy submodule; The clarification mechanism is: to output preset clarification scripts or provide an interface for selecting emotional tags; The conservative strategy is to respond in a neutral tone, avoid emotional expressions, and increase interactive options. S4-2-3, if there is moderate uncertainty, i.e., U low <U-score≤U high A compromise strategy is adopted, namely, adding probing statements to the standard response; S5, combining the interaction strategy adjusted in S4 with the user's question intent, uses the response generation unit to generate and output a response, including text and voice responses, specifically: The text response is generated through template matching or a seq2seq model, and the wording is adjusted in conjunction with the emotional tone. The voice response is synthesized through a TTS engine and its speed and tone are adjusted according to the emotion matching strategy.

5. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 4, characterized in that, The text data in S1-1 includes chat input and historical conversations; the voice data includes real-time voice streams with a sampling rate of ≥16kHz; and the visual data includes facial images or video streams with a resolution of ≥640×480 for key areas of the eyes and mouth.

6. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 4, characterized in that, The threshold validation set optimization in S4-1 maximizes interaction satisfaction; when the sentiment prediction result in S4-2-1 is identified as confused, the response is to call the detailed explanation document in the knowledge base.

7. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 4, characterized in that, The steps of the emotion clarification mechanism are as follows: Generate clarifying questions: Select the corresponding template based on the predicted sentiment category; Receive user feedback: Supports text input, emotion tag clicks, or voice confirmation; Feedback processing: If the user selects a specific emotion tag, update the current emotion status; if the user does not provide feedback, proceed to the conservative strategy process.

8. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 4, characterized in that, The uncertainty quantization training steps of the Bayesian neural network (BNN) are as follows: Data preparation: Collect a multimodal sentiment dataset containing text, speech, and images, and label discrete sentiment categories and continuous valence-arousal values; Network structure: Variational dropout is introduced into the output layers of the text encoder Transformer, the speech network CNN-LSTM, and the visual network ResNet to transform the deterministic network into a BNN; Loss function: Jointly optimizes the sentiment classification loss and uncertainty regularization term; Inference phase: The entropy value of the predicted distribution is calculated through N=20 MC samplings and used as the U-score.

9. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 8, characterized in that, Specifically, S4-1 involves using a grid search method to determine the threshold U. low and U high This includes the following steps: Define evaluation metrics: user subjective rating of interaction satisfaction, clarification request acceptance rate, and error sentiment response rate; Iterate through the threshold combinations and calculate the weighted score for each indicator; Choose the threshold combination that yields the highest weighted score as the final configuration.

10. The multimodal emotion recognition and interaction adjustment method based on uncertainty assessment according to claim 9, characterized in that, The threshold combination is U low ∈[0.2,0.4],U high ∈[0.6,0.8]).

Images