A multi-modal emotion recognition method and system based on confidence fusion

Abstract

The application discloses a multi-modal emotion recognition method and system based on confidence fusion, and belongs to the technical field of artificial intelligence. The application can model the uncertainty in multi-modal emotion data systematically, and reveal the dynamic changes of emotions through uncertainty estimation. It includes using MNE and OPENFACE toolkits to preprocess EEG signals and facial expression features respectively, using a multi-channel LSTM with shared weights to align the cross-modal features of the preprocessed multi-modal samples, and using a self-attention mechanism to model the uncertainty of sequence data on the time level. It also includes training two confidence regression modules to obtain the classification confidence of different modal feature representations as a guide for multi-modal fusion, weighting the feature representations of different modes based on the confidence obtained by the confidence regression module, optimizing the classification performance through cross-entropy loss, and improving the robustness of the model through self-paced learning.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a multimodal emotion recognition method and system based on confidence fusion. Background Technology

[0002] Emotion recognition is receiving increasing attention because it provides crucial information for perception and decision-making in human-computer interaction systems, enabling these systems to respond appropriately to users' emotional states. Among physiological emotion characteristics, electroencephalography (EEG) offers advantages such as being non-invasive, low-cost, and having high temporal resolution. Furthermore, since EEG signals are directly related to higher cognitive processes, EEG-based emotion recognition has gained increasing attention in recent years. However, the collection and recognition of emotion data are often affected by varying degrees of uncertainty. Therefore, developing robust recognition models that can effectively extract intrinsic emotional states remains a challenging task.

[0003] Many machine learning or deep learning-based methods have been used for EEG-based emotion recognition. HSAN can jointly model the local and global temporal information of EEG features. MNSEP, by learning the discriminant graph topology in the EEG brain network, can perform multi-class emotion recognition using multiple emotion-related spatial network topology patterns. OGSSL unifies adaptive graph learning and emotion recognition into a single objective. However, the above methods rely on unimodal analysis, which cannot fully reflect emotional states and is difficult to achieve satisfactory accuracy and stability. Benefiting from the complementary information between multimodal data, integrating physiological and behavioral signals can improve the performance of emotion recognition. ETF, based on the attention mechanism, fuses EEG and eye-tracking data to achieve high emotion recognition accuracy. MMResLSTM can learn the potential correlation between EEG and peripheral physiological signals, thereby improving recognition performance. EmotionMeter, by combining EEG and eye-tracking, integrates internal cognition and external subconscious behavior, exploring the complementary features of multimodal representation capabilities. However, the complex physiological characteristics of emotions make uncertainty a significant factor in emotion recognition. Specifically, the existence of various forms of uncertainty in multimodal data increases the difficulty of accurately identifying emotions.

[0004] In recent years, researchers have proposed several emotion recognition methods based on uncertainty learning. For example, EMENet captures uncertainty from multiple emotion descriptors using a deep ensemble model and applies iterative self-distillation to improve the performance of emotion recognition and uncertainty estimation. IQI can be used to describe the uncertainty of signal intensity in different frequency bands, thereby accurately extracting and selecting EEG emotion features. SCN uses a self-attention mechanism to weight each sample in training. It is worth noting that these uncertainty analysis methods are all developed for unimodal emotion prediction and cannot address the greater uncertainty in multimodal emotion data and fusion models. The uncertainty in multimodal emotion data and models can be summarized in three aspects: First, multimodal emotion data often have different semantic representations, and due to different mechanisms of emotion generation, the importance of each time window to the emotional state also varies, reflecting uncertainty at the temporal level. Second, the decision ambiguity generated by different modalities leads to uncertainty at the fusion level. Third, the difficulty or confidence level of the same model in recognizing emotion samples from different subjects or trajectories varies, reflecting uncertainty at the sample level. These uncertainties collectively pose challenges to multimodal emotion recognition. Therefore, there is an urgent need to develop effective fusion methods to mitigate the aforementioned uncertainties and improve the performance of multimodal emotion recognition. Summary of the Invention

[0005] Objective of the Invention: Addressing the shortcomings of single-modal EEG emotion recognition methods, the first objective of this invention is to provide a multimodal emotion recognition method based on confidence fusion. This method can systematically model the uncertainty in multimodal emotion data and reveal the dynamic changes in emotions through uncertainty estimation. Based on the application of this method, the second objective of this invention is to provide a multimodal emotion recognition system based on confidence fusion.

[0006] Technical Solution: A multimodal emotion recognition method based on confidence fusion. This method models the uncertainty in multimodal emotion data and reveals the dynamic changes of emotion through uncertainty estimation. The method includes the following steps:

[0007] S1. Obtain multimodal sentiment sequence data, preprocess the data, and obtain the preprocessed synchronous multimodal sequence matrix;

[0008] S2. Based on the preprocessed data obtained in step 1, the data is input into a multi-channel LSTM with shared weights to perform multimodal feature alignment and temporal feature extraction, thereby mining the potential correlations in the multimodal sequence data.

[0009] EEG signals and facial expression data are defined as follows: Where T represents the number of time windows for the data, and D E and D Fy represents the feature dimensions of the EEG modality and the facial expression modality, respectively, and the corresponding label information is denoted as y. To utilize the potential temporal dependence of continuous stimuli between EEG and facial expressions, a multi-channel LSTM network with shared weights (MC-LSTM represents the LSTM network process) is employed. Through shared weights, the multi-channel LSTM learns the correlations between modalities, as shown in the following formula:

[0010]

[0011] In the formula, and The memory cell, hidden state, and input data of the i-th sample in the LSTM for the (t-1)-th or t-th time window representing the modality m = {E, F}; Represents the output of a multi-channel LSTM. These are the memory units and hidden states of the t-th time window; the weights of the hidden states are shared between the modal and the time window, while the weights of the input vector are shared only between the modal and the time window; multi-channel LSTM can capture the potential correlations in multi-modal sequence data and achieve heterogeneous alignment;

[0012] S3. Use a self-attention mechanism to model time-level uncertainty, identify more reliable sequence data within time periods, and finally obtain the embedding features of the two modalities.

[0013] X' generated by multi-channel LSTM E and X' F Perform linear mapping transformations respectively:

[0014]

[0015]

[0016] In the formula, query Q m , key K m Sum V m The three sets of linear transformation features representing mode m are used to represent the mode m. and These are the query, key, and value parameter matrices used to generate the feature representation of modality m = {E, F}, which are updated through backpropagation during model training.

[0017] S4. A confidence regression network is designed based on the true class probability to obtain the confidence of the modality. At the same time, in order to maintain the common features among multimodal data, KL-divergence is used to constrain the similarity of different modalities.

[0018] The true class probability uses the SoftMax output probability corresponding to the true label as the prediction confidence. The TCP representation for each modality is as follows:

[0019] TCP m (x m ,y)=P(Y=y∣w,x m )

[0020] In the formula, x m This represents the high-dimensional feature vector of modality m, where y represents its true class. (TCP) m Then w represents the true class probability corresponding to the sample in modality m;

[0021] S5. Based on confidence level as a guide for multimodal fusion, reliable multimodal fusion is achieved. EEG and facial expression modal features are extracted through a self-attention mechanism, denoted as f respectively. E ∈R T×D f F ∈R T×D During training, a confidence regression network is used to dynamically estimate the prediction confidence of each modality, and this confidence is used as a weighted guide for the fusion of different emotional modalities.

[0022] S6. Classify based on the fused multimodal features to achieve emotion recognition, including using cross-entropy loss to supervise multimodal learning;

[0023] S7. In the optimization phase, self-paced learning is used to model the uncertainty at the sample level. In order to learn a robust representation of multimodal inputs, self-paced learning is used to train the above model based on samples with low to high uncertainty, thereby improving the robustness of the model.

[0024] Furthermore, step S1 specifically includes the following processes:

[0025] S1.1 The Python-based MNE toolkit and the MATLAB-based OPENFACE toolbox preprocess the dataset used, and perform downsampling on the EEG signals in the dataset, downsampling the signals to 128Hz;

[0026] S1.2 Use mne.filter() to perform bandpass filtering on the EEG signal to filter out frequency ranges in the EEG signal that are not related to emotional activation;

[0027] S1.3 Extract the power spectral density features of the t-second non-overlapping time window in the EEG using the Welch method;

[0028] S1.4 For facial expression data in the dataset, import the original video data in the dataset into the GUI interface of the OPENFACE toolbox;

[0029] S1.5 Use OPENFACE to extract features from facial videos, including facial position relative to the camera, head position, eye gaze direction, and facial motion units.

[0030] S1.6. Based on the t-second non-overlapping sliding window, take the average value of each feature of the time window as the final feature used.

[0031] In the method, the embedding feature extraction of the model in step S3 is as follows:

[0032] By calculating Q m and K m The attention coefficients between time windows are obtained by correlating the scalar dot product between them and then inputting them into the SoftMax function. Finally, the attention coefficients are correlated with V. m The product yields a self-attentional feature:

[0033]

[0034] In the formula, d represents the normalization parameter, which is related to K. m The dimensions are equal. K represents m The transpose of .

[0035] In the method, step S4, for mode m, uses a confidence regression network g m To approximate the true class probability, the confidence regression network is trained using MSE loss:

[0036]

[0037] Here, c m =g m (x m ) represents the confidence value predicted by the confidence regression network, and M is the number of modes. This represents the true value of TCP, and the true probability can be approximated by confidence regression networks.

[0038] Furthermore, in step S4, KL-divergence is used to constrain the similarity of different modalities, which not only allows for the learning of confidence level c m Furthermore, the predicted distribution of each modality can be obtained based on the SoftMax output of the proposed confidence regression network.

[0039] The KD-loss, calculated from the intermodal KL divergence, is expressed as:

[0040]

[0041]

[0042] here This represents the calculation of KL divergence.

[0043] In the method, step S5 further includes enhancing the destructiveness of modes with lower uncertainty by assigning higher weights; and suppressing modes with higher uncertainty by assigning lower weights, as mathematically represented below:

[0044]

[0045] Here c E c F These represent the confidence scores obtained from the confidence regression network for the EEG modality and the facial expression modality, respectively. Representative feature splicing operation.

[0046] In the method described, step S6 uses cross-entropy loss to supervise multimodal learning and avoid overfitting of the model. The calculation expression is as follows:

[0047]

[0048] In the formula, P(f) represents the cross-entropy loss. joint Y represents the probability of classifying a fused feature vector into a certain class, and Y is the label of the sample.

[0049] In the method described, the self-determined rhythm learning in step S7, based on samples with low to high uncertainty, is trained as follows:

[0050] Introduce sample weight variables v = [v1, ..., v n ] T And based on joint training loss As a measure of sample uncertainty, where α and β are hyperparameters controlling the loss weights, it is achieved by minimizing the following function:

[0051]

[0052] Here, n represents the number of training samples, and y i The label represents the sample.

[0053] The self-learning regularization term f(v) in the above formula i The regularization term λ is beneficial for selecting samples with low uncertainty from the training set. In this method, a hard regularization term is used, as follows:

[0054] f(v i ,λ)=-λv i

[0055] Then we can obtain an approximate solution corresponding to the above objective:

[0056]

[0057] The initial value of λ is 1, and the increment step of λ during training is 1.15.

[0058] Furthermore, in the initial stage, a small subset of training data is randomly selected to train the model; as training progresses, more samples are added to the training process based on the training loss of the samples, until all samples are included, which enables the model to learn more robust multimodal feature representations.

[0059] By implementing the above methods and steps, a multimodal emotion recognition system based on confidence fusion can be obtained. The system includes a multimodal feature alignment module based on time uncertainty, a confidence perception fusion module based on modal uncertainty, a classification module, and a model robustness optimization module based on sample uncertainty.

[0060] In this system, the confidence-aware fusion module consists of a multi-channel LSTM, a self-attention mechanism, a confidence regression module, and a self-stepping learning mechanism;

[0061] The multi-channel LSTM is used to perform cross-modal alignment of heterogeneous multimodal sentiment data, thereby better addressing the negative impact of modal heterogeneity on fusion.

[0062] The self-attention mechanism described above is used to exploit the temporal uncertainty of sequence data and further learn effective feature representations;

[0063] The confidence regression module is used to obtain the modal confidence of different modal features as a weighted guide for multimodal fusion, thereby improving the reliability of multimodal fusion and the accuracy of emotion recognition.

[0064] The self-stepping learning mechanism is used to allow the model to gradually incorporate uncertain samples into the model training process, thereby better modeling uncertainties at the sample level and improving the robustness and generalization ability of the model.

[0065] Beneficial effects: Compared with the prior art, the significant effects and substantial features of this invention are mainly as follows:

[0066] (1) This invention proposes a novel automated multimodal emotion recognition method that can systematically model the uncertainty in multimodal emotion recognition, realize the reliable fusion of multimodal emotion data, and clearly reveal the dynamic changes of emotions.

[0067] (2) A multi-channel LSTM feature extraction network with an attention mechanism was developed, which can simultaneously calibrate heterogeneous multimodal data and adaptively predict temporal uncertainty.

[0068] (3) A confidence regression module based on true class probability is proposed to estimate the uncertainty of emotion prediction at the modal level. Through confidence weighting, better fusion interpretability and more reliable multimodal emotion recognition performance are achieved.

[0069] (4) In the optimization process of the model, self-stepping learning is used to improve the robustness of the proposed model. Experimental results on multiple multimodal emotion datasets show that the proposed method outperforms the current state-of-the-art emotion recognition algorithms, and the effectiveness of the proposed method is verified through ablation studies. Attached Figure Description

[0070] Figure 1 This is a flowchart of the method described in this invention;

[0071] Figure 2 This is a flowchart of the data preprocessing process described in this invention;

[0072] Figure 3 This is a system framework diagram showing the application of the method described in this invention;

[0073] Figure 4 This is a framework diagram of the confidence regression module in this invention. Detailed Implementation

[0074] To illustrate the technical solutions disclosed in this invention in detail, further description is provided below with reference to the accompanying drawings.

[0075] Existing unimodal EEG emotion recognition methods struggle to reflect the emotional state of subjects and fail to utilize complementary information from multimodal data, resulting in unsatisfactory accuracy and stability. Furthermore, the complex physiological characteristics of emotions make uncertainty a significant factor in emotion recognition, and the various forms of uncertainty in multimodal emotion recognition further complicate accurate identification.

[0076] This invention provides a multimodal emotion recognition method based on confidence fusion, combining... Figure 1 The illustrated process uses a fully connected neural network layer. The multi-channel LSTM network with shared weights has 256 hidden layers (1 layer total), a self-attention linear transformation layer with 256 nodes, and a feedforward network with 512 hidden layers. Softmax is used as the activation function in the last layer, and ReLU is used throughout the model. The implementation steps of the method described in this invention are further described below:

[0077] S1: Obtain multimodal sentiment sequence data, preprocess the data, and obtain the preprocessed synchronous multimodal sequence matrix.

[0078] Combination Figure 2 As shown, the data preprocessing in step S1 is as follows:

[0079] The datasets used were preprocessed using the Python-based MNE toolkit and the MATLAB-based OPENFACE toolkit.

[0080] For the EEG signal in the dataset, the signal is first downsampled to 128Hz.

[0081] Use mne.filter() to apply a bandpass filter to the EEG signal to filter out frequency ranges in the EEG signal that are not related to emotional activation.

[0082] Next, we use mne.time_frequency.psd_welch, which employs the Welch method, to extract the power spectral density features of the t-second non-overlapping time window in the EEG.

[0083] For the facial expression data in the dataset, first open the GUI interface of the OPENFACE toolbox and import the original video data from the dataset.

[0084] Next, features of the facial video were extracted using OPENFACE, including 3 facial positions relative to the camera, 3 head positions, 6 eye gaze directions, and 17 facial motion units.

[0085] Finally, similar to the EEG signal, the average value of each feature in the time window is taken as the final feature used, based on a t-second non-overlapping sliding window.

[0086] S2. Based on the preprocessed data obtained in step 1, the data is input into a multi-channel LSTM with shared weights to perform multimodal feature alignment and temporal feature extraction, thereby mining the potential correlations in the multimodal sequence data.

[0087] EEG signals and facial expression data are defined as follows: Where T represents the number of time windows for the data, and D E and D F y represents the feature dimensions of the EEG modality and the facial expression modality, respectively, and their corresponding label information is denoted as y. To utilize the potential temporal dependence of continuous stimuli between EEG and facial expressions, this invention employs a multi-channel LSTM network with shared weights (MC-LSTM). By sharing weights, the multi-channel LSTM can learn the correlations between modalities. The specific formula is as follows:

[0088]

[0089] here and The memory cell, hidden state, and input data of the i-th sample in the LSTM for the (t-1)-th or t-th time window representing the mode m = {E, F}. Represents the output of a multi-channel LSTM. This represents the memory unit and hidden state of the t-th time window. The weights of the hidden state are shared between the modality and the time window, while the weights of the input vector are shared only between the modality and the time window. Multichannel LSTM can capture latent correlations in multimodal sequence data and achieve heterogeneous alignment.

[0090] Step 3: Use a self-attention mechanism to model temporal uncertainty, identify more reliable sequence data within a time period, and finally obtain the embedding features of the two modalities.

[0091] Due to differences in emotional patterns, even under prolonged and continuous stimuli, it is difficult to guarantee consistent emotional predictions across different patterns. This leads to potential uncertainty in certain time periods of physiological or non-physiological sequences. To overcome this limitation and enhance the specificity of temporal information, this invention employs a self-attention mechanism to identify more reliable time periods in sequence data, specifically X' generated by a multi-channel LSTM. E and X' F Perform linear mapping transformations respectively:

[0092]

[0093]

[0094] The query Q here m , key K m Sum V m The three sets of linear transformation features represent mode m. and These are the query, key, and value parameter matrices used to generate the feature representation of modality m = {E, F}, which are updated via backpropagation during model training. Q is calculated... m and K m The attention coefficients between time windows are obtained by correlating the scalar dot product between them and then input into the SoftMax function.

[0095] Then, by using the attention coefficient and V m The product yields a self-attentional feature:

[0096]

[0097] Here, d represents the normalization parameter, which is related to K. m The dimensions are equal. K represents m The transpose of .

[0098] Step 4: Design a confidence regression network based on the true class probability to obtain the confidence of the modality.

[0099] While multimodal fusion typically enhances emotion recognition performance due to the complementarity of multimodal information, inconsistencies in decision-making across different modalities can negatively impact its effectiveness. Therefore, estimating the prediction confidence of each modality is crucial for effectively integrating multimodal information. Higher prediction confidence corresponds to lower uncertainty in the model's predictions. Consequently, this invention introduces the true class probability (TCP) as a metric for measuring modality confidence.

[0100] TCP uses the SoftMax output probability corresponding to the real label as the prediction confidence. Formally, TCP for each modality can be represented as:

[0101] TCP m (x m ,y)=P(Y=y∣w,x m )

[0102] Here x m This represents the high-dimensional feature vector of modality m, where y represents its true class. (TCP) m This represents the true class probability corresponding to the sample in modality m. Although TCP can obtain reliable confidence scores, it cannot be used directly during the testing phase due to the lack of label information.

[0103] For mode m, this invention designs a confidence regression network g. m To approximate TCP, we use MSE loss to train a confidence regression network:

[0104]

[0105] Here, c m =g m (x m ) represents the confidence value predicted by the confidence regression network, and M is the number of modes. This represents the true value of TCP. TCP can then be approximated using a confidence regression network. Simultaneously, to preserve common features among multimodal data, KL-divergence is used to constrain the similarity between different modalities. This not only allows learning the confidence level c... m Furthermore, the predicted distribution of each modality can be obtained based on the SoftMax output of the proposed confidence regression network. Therefore, the KD-loss calculated from the intermodal KL divergence can be expressed as:

[0106]

[0107]

[0108] here This represents the calculation of KL divergence.

[0109] Step 5: Based on confidence level as a guide for multimodal fusion, achieve trusted multimodal fusion.

[0110] This method utilizes a self-attention mechanism to extract EEG and facial expression modal features, denoted as f, respectively. E ∈R T×D f F ∈R T×D During training, a confidence regression network is used to dynamically estimate the prediction confidence of each modality, and this confidence is used as a weighting guide for the fusion of different emotional modalities. By assigning higher weights, the destructiveness of modalities with lower uncertainty can be enhanced; by assigning lower weights, the destructiveness of modalities with higher uncertainty can be suppressed.

[0111]

[0112] Here c E c F These represent the confidence scores obtained from the confidence regression network for the EEG modality and the facial expression modality, respectively. This represents the feature concatenation operation. The model models uncertainty at the modal level, making multimodal fusion predictions more reliable.

[0113] Step 6: Classify based on the fused multimodal features to achieve emotion recognition.

[0114] For multimodal tasks, existing techniques typically use binary cross-entropy (BCE) to supervise the learning of each branch, but this severely penalizes samples that cannot be classified from a particular modality. In this case, the model may overfit biases in the data, leading to overfitting. Therefore, based on the joint feature representation obtained by confidence-aware fusion, this invention uses cross-entropy (CE) loss to supervise multimodal learning:

[0115]

[0116] Here P(f) joint Y represents the probability of classifying a fused feature vector into a certain class, and Y is the label of the sample.

[0117] Step 7: In the optimization phase, self-stepping learning is used to model uncertainties at the sample level to improve the robustness of the model.

[0118] If samples with high training loss are introduced too early in the training process, they are more likely to negatively impact the robustness of the model. To learn robust representations of multimodal inputs, this method employs self-paced learning, training the model based on samples with varying degrees of uncertainty, from low to high.

[0119] Specifically, sample weight variables v = [v1, ..., v] are introduced. n ] T And based on joint training loss As a measure of sample uncertainty, where α and β are hyperparameters controlling the loss weights. This is achieved by minimizing the following function:

[0120]

[0121] Here, n represents the number of training samples, and y i The label represents the sample. The self-learning regularization term f(v) in the above formula... i The expression λ is beneficial for selecting samples with low uncertainty from the training set. In the method described in this invention, a hard regularization term is employed, expressed as follows:

[0122] f(v i ,λ)=-λv i

[0123] Then we can obtain an approximate solution corresponding to the above objective:

[0124]

[0125] The initial value of λ is 1, and the increment step of λ during training is 1.15. In the initial stage, this invention randomly selects a small portion of the training data to train the model. As training progresses, more samples are added to the training process based on the training loss of the samples, until all samples are included. This enables the model to learn more robust multimodal feature representations.

[0126] Further integration Figure 3 and Figure 4 The multimodal emotion recognition model proposed in this invention includes a multimodal feature alignment module based on temporal uncertainty, a confidence-aware fusion module based on modal uncertainty, a classification module, and a model robustness optimization module based on sample uncertainty. The overall framework structure of the proposed system is as follows: Figure 1 As shown, the framework structure of the confidence regression module is as follows: Figure 4As shown, it consists of a multi-channel LSTM, a self-attention mechanism, a confidence regression module, and a self-stepping learning mechanism. The multi-channel LSTM performs cross-modal alignment on heterogeneous multimodal emotion data, thus better addressing the negative impact of modal heterogeneity on fusion. The self-attention mechanism can exploit the temporal uncertainty of sequence data to further learn effective feature representations. The confidence regression module obtains the modal confidence of features from different modalities as a weighted guide for multimodal fusion, thereby improving the reliability of multimodal fusion and the accuracy of emotion recognition. The self-stepping learning mechanism allows the model to gradually incorporate uncertain samples into the model training process, thereby better modeling sample-level uncertainty and improving the model's robustness and generalization ability. Conf The purpose of this approach is to address the issue that the true class probability, which requires label information, cannot be calculated during testing. By training the confidence regression module, the model can approximate the true class probability value during the testing phase. Kullback-Leibler Divergence Loss (L...) KD Cross-Entropy Loss (L) is used to constrain feature similarity between different modalities, thereby preserving common features among multimodal data and avoiding the loss of key emotional features. Furthermore, Cross-Entropy Loss (L) is used to constrain feature similarity between different modalities, thus preserving common features among multimodal data and avoiding the loss of key emotional features. CE The purpose of this is to train the classifier and improve the model's emotion recognition performance by narrowing the distance between the predicted label and the real label.

[0127] In summary, the multi-channel LSTM in the method provided by this invention performs cross-modal alignment on heterogeneous multimodal emotion data, thereby better addressing the negative impact of modal heterogeneity on fusion. The self-attention mechanism can exploit the temporal uncertainty of sequence data, further learning effective feature representations. The confidence regression module obtains the modal confidence of different modal features as a weighted guide for multimodal fusion, thereby improving the reliability of multimodal fusion and the accuracy of emotion recognition. The self-stepping learning mechanism allows the model to gradually incorporate uncertain samples into the model training process, thereby better modeling sample-level uncertainty and improving the model's robustness and generalization ability. Conf The purpose is to address the issue that the true class probability, which requires label information, cannot be calculated during testing. By training the confidence regression module, the model can approximate the true class probability value during the testing phase. KD This is used to constrain the feature similarity between different modalities, thereby preserving common features among multimodal data and avoiding the loss of key emotional features. Furthermore, Cross-Entropy Loss (L...) CEThe purpose of this is to train the classifier and improve the model's emotion recognition performance by narrowing the distance between the predicted label and the real label.

Claims

1. A multimodal emotion recognition method based on confidence fusion, characterized in that, This method models the uncertainty in multimodal sentiment data and reveals the dynamic changes in sentiment through uncertainty estimation, including the following steps: S1. Obtain multimodal sentiment sequence data, preprocess the data, and obtain the preprocessed synchronous multimodal sequence matrix; S2. Based on the preprocessed data in step 1, the data is input into a multi-channel LSTM with shared weights to perform multimodal feature alignment and temporal feature extraction, thereby mining the potential correlations in the multimodal sequence data. EEG signals and facial expression data are defined as follows: , ,in The number of time windows representing the data. and These represent the feature dimensions of the EEG modality and the facial expression modality, respectively, and the corresponding label information is denoted as... To leverage the potential temporal dependence of continuous stimuli between EEG and facial expressions, a multi-channel LSTM network with shared weights is employed. Through weight sharing, the multi-channel LSTM learns the correlations across modalities, as expressed in the following formula: In the formula, and Representative mode The The first sample or the The memory cells, hidden states, and input data of each time window in the LSTM; Represents the output of a multi-channel LSTM. Then it is the first The memory cells and hidden states of each time window; the weights of the hidden states are shared between the modal and the time window, while the weights of the input vector are shared only between the modal and the time window; the multi-channel LSTM can capture the potential correlations in multimodal sequence data and achieve heterogeneous alignment; S3. Use a self-attention mechanism to model time-level uncertainty, identify more reliable sequence data within time periods, and finally obtain the embedding features of the two modalities. Generated by multi-channel LSTM and Perform linear mapping transformations respectively: In the formula, query ,key Sum Representative mode The three sets of linear transformation characteristics are represented as follows: , and These are used to generate modes. The parameter matrix representing the query, key, and value is updated via backpropagation during model training; S4. A confidence regression network is designed based on the true class probability to obtain the confidence of the modality. At the same time, in order to maintain the common features among multimodal data, KL-divergence is used to constrain the similarity of different modalities. The true class probability uses the SoftMax output probability corresponding to the true label as the prediction confidence. The TCP representation for each modality is as follows: In the formula, Representing modes The high-dimensional feature vector of the sample Represents its true category, This represents a mode. The true class probability corresponding to the sample. Represents the weights of the neural network; S5. Based on confidence level as a guide for multimodal fusion, reliable multimodal fusion is achieved. EEG and facial expression modal features are extracted through a self-attention mechanism, and are represented as follows: , During training, a confidence regression network is used to dynamically estimate the prediction confidence of each modality, and this confidence is used as a weighted guide for the fusion of different emotional modalities. S6. Classify based on the fused multimodal features to achieve emotion recognition, including using cross-entropy loss to supervise multimodal learning; S7. In the optimization stage, self-stepping learning is used to model the uncertainty at the sample level. In order to learn a robust representation of multimodal inputs, a self-stepping learning mechanism is used and the above model is trained based on samples with low to high uncertainty, thereby improving the robustness of the model. The self-step learning is based on training with samples ranging from low to high uncertainty, and the process is as follows: Introducing sample weight variables And based on joint training loss As a measure of sample uncertainty, where, Represents cross-entropy loss, It is an MSE loss. This represents the calculation of KL divergence between modes. and It is a hyperparameter that controls the loss weights; Self-tracking learning performs joint training by minimizing a function: In the formula, Represents the number of training samples. Representing the Weight variables for each training sample; The regularization term in the above formula This method facilitates the selection of low-uncertainty samples from the training set. A hard regularization term is employed, as follows: Then we can obtain an approximate solution corresponding to the above objective: Among them, initialization The value is 1. The incremental step size during training is 1.15; In the initial stage, a small subset of training data is randomly selected to train the model; as training progresses, more samples are added to the training process based on the training loss of the samples, until all samples are included, which enables the model to learn more robust multimodal feature representations.

2. The multimodal emotion recognition method based on confidence fusion according to claim 1, characterized in that, Step S1 specifically includes the following process: S1.1 The Python-based MNE toolkit and the MATLAB-based OPENFACE toolbox preprocess the dataset used, and perform downsampling processing on the EEG signals in the dataset; S1.2 Use mne.filter() to perform bandpass filtering on the EEG signal to filter out frequency ranges in the EEG signal that are not related to emotional activation; S1.3 Extract the power spectral density features of the t-second non-overlapping time window in the EEG signal using the Welch method; S1.4 For facial expression data in the dataset, import the original video data in the dataset into the GUI interface of the OPENFACE toolbox; S1.5 Use OPENFACE to extract features from facial videos, including facial position, head position, eye gaze direction, and facial motion units; S1.

6. Based on the t-second non-overlapping sliding window, take the average value of each feature of the time window as the final feature used.

3. The multimodal emotion recognition method based on confidence fusion according to claim 1, characterized in that, The embedding feature extraction of the model described in step S3 is as follows: Calculate the query s and keys The attention coefficients between time windows are obtained by correlating the scalar dot product between them and then inputting them into the SoftMax function. The attention coefficients are then compared with... The product yields a self-attentional feature: In the formula, Represents the normalization parameter, which is related to... The dimensions are equal. express The transpose of .

4. The multimodal emotion recognition method based on confidence fusion according to claim 1, characterized in that, In step S4, for the mode Through a confidence regression network To approximate the true class probability, the confidence regression network is trained using MSE loss: Here This represents the confidence value predicted by the confidence regression network. It is the number of modes. This represents the true value of TCP, and the true probability can be approximated by confidence regression network.

5. The multimodal emotion recognition method based on confidence fusion according to claim 1 or 4, characterized in that, In step S4, KL-divergence is used to constrain the similarity of different modalities, which not only allows for the learning of confidence levels. Furthermore, the predicted distribution of each modality can be obtained based on the SoftMax output of the proposed confidence regression network. ; The KD-loss, calculated from the intermodal KL divergence, is expressed as: here This represents the calculation of KL divergence.

6. The multimodal emotion recognition method based on confidence fusion according to claim 1, characterized in that, Step S5 also includes enhancing the destructiveness of modes with lower uncertainty by assigning higher weights, and suppressing modes with higher uncertainty by assigning lower weights, as mathematically represented below: here , These represent the confidence scores obtained from the confidence regression network for the EEG modality and the facial expression modality, respectively. Representative feature splicing operation.

7. The multimodal emotion recognition method based on confidence fusion according to claim 1, characterized in that, Step S6 uses cross-entropy loss to supervise multimodal learning and avoid overfitting of the model. The calculation expression is as follows: In the formula, Represents cross-entropy loss, The probability of classifying a feature vector into a certain category by fusing the feature vectors. The label for the sample.

8. A multimodal emotion recognition system based on confidence fusion, the system being used to execute the multimodal emotion recognition method based on confidence fusion as described in claim 1, characterized in that, The system includes a multimodal feature alignment module based on time uncertainty, a confidence-aware fusion module based on modal uncertainty, a classification module, and a model robustness optimization module based on sample uncertainty. Furthermore, the confidence-aware fusion module consists of a multi-channel LSTM, a self-attention mechanism, a confidence regression module, and a self-stepping learning mechanism; The multi-channel LSTM is used to perform cross-modal alignment of heterogeneous multimodal sentiment data, thereby better addressing the negative impact of modal heterogeneity on fusion. The self-attention mechanism described above is used to mine the temporal uncertainty of sequence data and further learn effective feature representations; The confidence regression module is used to obtain the modal confidence of different modal features as a weighted guide for multimodal fusion, thereby improving the reliability of multimodal fusion and the accuracy of emotion recognition. The self-stepping learning mechanism is used to allow the model to gradually incorporate uncertain samples into the model training process, thereby better modeling uncertainties at the sample level and improving the robustness and generalization ability of the model.

Images