Multimodal emotion recognition method and system based on hypergraph diffusion and evidence fusion, terminal and storage medium

Abstract

The application relates to the technical field of image analysis, and discloses a multi-modal emotion recognition method, system, terminal and storage medium based on hypergraph diffusion and evidence fusion, the method comprising the following steps: simulating the case of random data loss by randomly generating a mask to randomly mask a data set, using a trained conditional diffusion model to inversely sample the preprocessed simulation data, obtaining a training set with the missing modalities completed, training an emotion classification network, and finally performing emotion recognition. Through double-channel evidence fusion, the application simultaneously estimates uncertainty at the feature source level and the discrimination level, thereby realizing adaptive evidence fusion, reducing the performance decline caused by modal loss, explicitly recovering the potential features of the missing modalities in the feature space, and improving the accuracy of the final emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to a multimodal emotion recognition method, system, terminal, and computer-readable storage medium based on hypergraph diffusion and evidence fusion. Background Technology

[0002] Multimodal Emotion Recognition in Conversations (MERC) has become a key technology in affective computing. This technology accurately identifies human emotions through the collaborative fusion of multimodal signals, including text, acoustic, and visual signals.

[0003] However, in existing real-world scenarios, the unavoidable loss of modality often affects the performance of multimodal understanding, leading to poor model interpretability, feature confusion, and semantic distortion. For methods that cannot recover modalities, there may be insufficient information, making it difficult for the model to compensate for key semantics from the remaining modalities, resulting in a significant performance drop, confusion of features from different modalities, and a decrease in the accuracy of emotion recognition results.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a multimodal emotion recognition method, system, terminal, and computer-readable storage medium based on hypergraph diffusion and evidence fusion. This invention aims to address the problems in existing technologies where recoverable modal methods for emotion recognition rely excessively on complete data, and where non-recoverable modal methods may suffer from insufficient information, leading to inaccurate emotion recognition results.

[0006] To achieve the above objectives, this invention provides a multimodal emotion recognition method based on hypergraph diffusion and evidence fusion, which includes the following steps:

[0007] Multiple training statements and multiple modalities of each training statement are obtained. A hypergraph network is constructed using mask vectors. The set of nodes and the set of hyperedges in the hypergraph network are aggregated to update all nodes in the hypergraph network and obtain the corresponding node representations.

[0008] All node representations are concatenated according to modal dimension to obtain conditional information. The conditional information is input into the conditional diffusion model to obtain multiple output features. A fusion feature is constructed based on all the output features.

[0009] Inverse sampling is performed based on the conditional information to fill in all missing modes and generate mean squared error and training set;

[0010] Based on the mean squared error, calculate the feature uncertainty of the entire training set; based on the class probability simplex, calculate the discrimination uncertainty of the entire training set; and construct a fused probability vector based on the feature uncertainty and the discrimination uncertainty.

[0011] Multiple uncertainty estimates are determined, a divergence regularization term is constructed, the emotion classification network is optimized based on all the uncertainty estimates, the divergence regularization term and the probability vector, and the test set is input into the optimized emotion classification network to output the emotion recognition result.

[0012] This invention achieves adaptive evidence fusion by simultaneously estimating uncertainty at both the feature source and discrimination levels through dual-channel evidence fusion. This mitigates the performance degradation caused by modality loss, explicitly recovers the latent features of missing modalities in the feature space, and improves the accuracy of the final emotion recognition. Attached Figure Description

[0013] Figure 1 This is a flowchart of a preferred embodiment of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion of the present invention;

[0014] Figure 2 This is a schematic diagram of the network structure of a preferred embodiment of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion of the present invention;

[0015] Figure 3 This is a visualization of attention score allocation during the aggregation process of a preferred embodiment of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion of the present invention;

[0016] Figure 4 This is a visualization of the recovered features and original features of a preferred embodiment of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion of the present invention;

[0017] Figure 5 This is a structural diagram of a preferred embodiment of the multimodal emotion recognition system based on hypergraph diffusion and evidence fusion of the present invention;

[0018] Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Non-modal recovery methods perform inference directly under the condition of missing input, without "reconstructing" the missing modal information. Instead, they improve the model structure or optimization strategies to enable the model to complete the task on incomplete multimodal input. Modal recovery methods estimate and reconstruct the features of missing modalities, allowing downstream tasks to "superficially" recover to the input conditions with complete multimodalities. Explicit modal recovery directly uses generative models to explicitly "reconstruct" the missing modalities. By learning the joint distribution of multimodalities and backsampling, it generates features that are as consistent as possible with the true modalities.

[0021] However, when the amount of missing modal information is large, the usable features extracted by distillation or correlation constraints alone are often insufficient to support high-precision classification or regression. This is especially true when there are significant semantic differences between multimodalities, or when certain modalities carry core task cues. If a modality is missing, the model struggles to compensate for the key semantics from the remaining modalities, leading to a significant performance degradation. Modal recovery methods, on the other hand, suffer from feature confusion, semantic distortion, and poor model interpretability. Explicitly recovered models require access to complete multimodal data during training to learn accurate joint distributions and mappings. If a modality has a "fixed missing" component in a local or global dataset, the generative model cannot obtain the true sample distribution of that modality from the original data, causing the training process to crash due to severe missing samples or resulting in serious distribution estimation bias.

[0022] The preferred embodiment of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion described in this invention, such as... Figure 1 As shown, the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion includes the following steps:

[0023] Step S10: Obtain multiple training statements and multiple modalities of each training statement, construct a hypergraph network using mask vectors, and aggregate the node set and hyperedge set in the hypergraph network to update all nodes in the hypergraph network and obtain the corresponding node representation.

[0024] In response to the aforementioned problems, this invention proposes a sentiment recognition framework based on hypergraph diffusion and evidence fusion (HyperEF, e.g., ...). Figure 2 As shown, this approach aims to alleviate the challenges posed by incomplete modalities in MERC. By using the semantic information extracted by the masked hypergraph attention mechanism as a condition for the conditional diffusion model, semantic ambiguity of the recovered features is reduced.

[0025] Specifically, a training dialogue is obtained, and multiple training statements and multiple modalities for each training statement are constructed based on the training dialogue:

[0026] ;

[0027] in, Indicates training dialogue, , and They represent the 1st, 2nd, and 3rd respectively. One training statement, , and They represent , and The statement object;

[0028] ;

[0029] in, Indicates the first One training statement, , and They represent The text modality, visual modality, and audio modality;

[0030] Define a corresponding mask for each of the aforementioned modalities, randomly define multiple of the aforementioned modalities as missing modalities, and embed the corresponding masks into the positions corresponding to all the missing modalities to construct a hypergraph network:

[0031] ;

[0032] in, This represents a hypergraph network. , , and They represent , , and The mask, , and They represent The masks corresponding to the text modality, visual modality, and audio modality;

[0033] ;

[0034] ;

[0035] in, This represents the set of nodes in a hypergraph network. and They represent the 1st and the 2nd respectively. 1 node Denotes the set of superedges in a hypergraph network. , and These represent the 1st, 2nd, and 3rd context edges, respectively. and They represent the 1st and the 2nd respectively. A multimodal edge;

[0036] ;

[0037] in, This indicates an embedding operation. Indicates the first A node after embedding a mask. Indicates the first Each node.

[0038] In existing graph networks, each edge connects only two nodes, making it difficult to capture higher-order relationships. Therefore, this invention proposes a Masked Hypergraph Attention (MHCAT) network. MHCAT not only captures missing modalities in the dataset through masking, but its hypergraph edges can connect any number of nodes, thus potentially capturing higher-order multivariate relationships in multimodal dialogue data. Furthermore, an attention mechanism is added to MHCAT, allowing dynamic adjustment of the weights between each node and hyperedge to capture accurate higher-order semantic information in real time, thereby guiding the subsequent denoising process of the conditional diffusion model.

[0039] In one embodiment of this invention, a training dialogue contains multiple training statements, each of which includes three modalities (text, visual, and audio). To address the random modality missingness in real-world scenarios, a mask is defined for each statement and its three modalities, with available modalities defined as 1 and missing modalities as 0, serving as prior input for subsequent graph modeling and conditional diffusion.

[0040] Furthermore, a hypergraph network is constructed based on these modalities and statements. Each node in the hypergraph network represents a modality, and the node set includes single-modal utterances (i.e., each statement corresponds to three modalities). The hyperedge set includes context edges and multimodal edges. Context edges connect all context nodes in a single modality, and multimodal edges connect all modalities in a single statement. During node construction, the positional relationships of each missing node are preserved, only its numerical value is set to zero. Then, an embedded mask vector is added to the corresponding node to simulate the missing modalities in the dataset using the mask vector.

[0041] Furthermore, since traditional hypergraph networks rely on a fixed association matrix, their static nature limits their adaptability during the aggregation process, making it difficult to focus on more informative relationships. Therefore, this invention adds an attention mechanism to the masked hypergraph network, employing a two-stage aggregation method when updating node representations, thereby dynamically adjusting the weights of each node and the variable weights.

[0042] Specifically, a hyperedge attention score is constructed. For each hyperedge, weights are assigned to each node on the hyperedge based on the hyperedge attention score. Then, each node on the hyperedge is aggregated to update the hyperedge representation, until all hyperedges are updated.

[0043] ;

[0044] in, Indicates the first The updated hyperedge representation after layer update Represents a non-linear activation function. Indicates the first superedge and the first The super-edge attention score between nodes Indicates the first The set of nodes for an edge. Indicates the first The first layer Each node represents a node. Indicates weight;

[0045] ;

[0046] in, Represents an exponential function. This represents the activation function. This represents a trainable weight vector. Indicates weight, This indicates a splicing operation. Indicates the first The updated hyperedge representation after layer update Indicates the first The first layer Each node represents a node. Indicates the first 1 node Indicates the first The first layer Each node represents a node;

[0047] Construct node attention scores. For each node, assign weights to all updated hyperedges connected to that node based on the node attention score. Then, aggregate all hyperedges to update the node, obtaining the node representation of that node. Repeat this process until all nodes have been updated.

[0048] ;

[0049] in, Indicates the first The updated node representation of the layer. Indicates the first A super edge, Indicates and The set of connected hyperedges Indicates the first The node and the first Attention scores between nodes along the superedges;

[0050] ;

[0051] in, This represents a trainable weight vector. Indicates the first The first layer Each hyperedge is represented.

[0052] In the first phase, the aggregation of each hyperedge is achieved by connecting nodes to hyperedges (where, Figure 2 In , , and The first The text modal nodes of the 1st, 2nd, 3rd, and 4th nodes of the layer are represented as follows: Indicates the first The visual modal node representation of the first node in the layer, Indicates the first The first node of the layer represents the audio modality node; DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a pre-trained language model; MA-Net (Multi-scale Attention Network) is a multi-scale attention network based on self-attention mechanism; Wav2Vec (Waveform-to-Vector) is a speech representation learning technique based on self-supervised learning. For each hyperedge, MHGAT first aggregates all nodes connected to this hyperedge to update its hyperedge representation (i.e., obtain the hyperedge of the next layer), and then uses an attention mechanism to assign weights to each node according to its contribution (e.g., ...). Figure 3 As shown in (a)). In this process, since the initial features of the hyperedge are missing, a learnable embedding vector is first defined to achieve the above operation.

[0053] Furthermore, for the second stage, aggregation of each node is achieved through hyperedges to nodes (where, Figure 2 In Indicates the first The text modal node representation of the first node in the layer, Indicates the first The first layer Audio modal node representation of each node. and Let the attention scores of the first hyperedge and the first node be represented respectively. superedge and the first The super-edge attention score between nodes and Then, these represent the hyperedge attention scores between the first node and the first hyperedge, and the scores between the first node and the first hyperedge, respectively. The superedge attention score between the node and the third superedge. Indicates the first (Multimodal edges). After the hyperedges are updated, all hyperedges connected to a node are aggregated to update that node. This aggregation process is also achieved through an attention mechanism, ultimately yielding the node representation for each node. Through a bidirectional aggregation mechanism, MHGAT can capture complex multivariate relationships in multimodal dialogues and, combined with a masking mechanism, distinguish missing modalities. The captured dialogue dependency information helps the diffusion model maintain consistency with the text modality when recovering missing visual or audio data, significantly reducing the semantic shift between recovered and original features. Simultaneously, the masking participates in graph construction and attention allocation, distinguishing between "available / missing / noise" sources and improving robustness under different missing rates and missing modes.

[0054] For example, in Figure 3 middle, Figure 3 (a) shows the "node-to-hyperedge" process for the text modality and the "hyperedge-to-node" process for the 8th node. From the "node-to-hyperedge" process, it's clear that in the context dimension of the text(t) modality, the model assigns higher attention scores to utterances containing strong sentiment words (such as "funny" and "like"), while giving lower scores to ambiguous expressions (such as "Okay"). Furthermore, when the text modality cannot provide sufficient semantic information, multimodal hyperedges supplement the necessary knowledge. For example, since the text "Okay" appears in both the 5th and 8th nodes, the model distinguishes their sentiment by assigning higher attention to the audio and visual modalities during multimodal hyperedge aggregation. Finally, in the "hyperedge-to-node" step for the 8th node (e.g.... Figure 3In (b) of the model, the attention score of the multimodal hyperedge (0.69) is higher than that of the context hyperedge (0.31). After introducing missing modality masking information, the three modalities in the context dimension tend to give higher attention to nodes 6, 7, and 9; while in the multimodal dimension, the model tends to assign lower attention to the missing modality. This shows that the masking mechanism can effectively capture the information of "whether a certain modality is missing". The source of decisive evidence can be located by visualizing the two-stage attention weights, and multi-scale cross-attention improves the consistency of generated details.

[0055] This invention uses dialogue-level high-order semantics extracted by "masked hypergraph attention" as conditional signals for the diffusion model, explicitly recovering missing modalities in the feature space. Unlike graph methods that model only two nodes, the hypergraph connects multiple nodes simultaneously with "hyperedges," combining masking to indicate possible / missing modalities. Attention aggregation is performed in two stages: "node → hyperedge" and "hyperedge → node," dynamically filtering more informative context and cross-modal relationships. Subsequently, conditions are injected with cross-attention at the multi-scale stages of the diffusion U-Net, and a classifier-free guidance strategy is used to robustly control the strength of the conditions, thereby generating semantically consistent missing modal representations even in the case of missing modalities. This approach differs from existing compensatory methods that only infer semantics in the fusion space and direct generative methods that lack high-order prior context.

[0056] Furthermore, in addition to the hypergraph network proposed in this embodiment, Graph Convolutional Network (GCN) can be used for message passing and aggregation, Graph Attention Network (GAT) can be used for attention-weighted aggregation of nodes, or GraphSAGE (an inductive learning method based on graph neural networks) can be used for sampling and aggregation to support offline training of large-scale graphs and achieve the technical effects achieved by the hypergraph network.

[0057] Step S20: Concatenate all the node representations according to the modal dimension to obtain conditional information, input the conditional information into the conditional diffusion model to obtain multiple output features, and construct a fusion feature based on all the output features.

[0058] In this process, the node representations updated by MHGAT are concatenated along the modal dimension to obtain conditional signals, which are then input into the conditional diffusion model. The conditional information is used in both the training and sampling phases to guide the diffusion model in generating restored features that are consistent with the semantics of the original features.

[0059] Specifically, based on the dimension of the modality represented by each node, all node representations are concatenated to obtain conditional information; this conditional information is input into a conditional diffusion model, which performs cross-attention fusion with the input features and the conditional information to obtain multiple output features; conditional information is randomly filtered out from the multiple output features with a fixed probability to obtain multiple unconditional output features; and all unconditional output features and the unfiltered output features are weighted and fused to obtain fused features.

[0060] ;

[0061] in, Indicates the first The fusion characteristics of step diffusion, The weights represent the feature fusion weights. This indicates conditional output features. Indicates unconditional output features. Indicates the first Step-diffusion samples, Indicates the number of diffusion steps. Indicates conditional information.

[0062] In another embodiment of the present invention, a U-Net structure is used as the backbone network of the diffusion network. In each upsampling and downsampling stage, the current feature and conditional information are fused by cross-attention. Then, in the subsequent training process, the conditional information is randomly filtered out with a fixed probability to obtain conditional output and unconditional output. The two are then fused together to obtain fused features for subsequent training.

[0063] Furthermore, the conditional diffusion model proposed in this embodiment can also use different convolutional kernel sizes (such as 3×3, 5×5) or different channel number allocations in the downsampling and upsampling branches of U-Net to adapt to the scale of different modal features; if real-time requirements are high, U-Net can be modified to a lightweight architecture to reduce the amount of computation.

[0064] Step S30: Perform inverse sampling based on the condition information to fill in all missing modes and generate mean squared error and training set.

[0065] Specifically, the actual noise and predicted noise generated during forward diffusion of the conditional diffusion model are extracted, and a noise objective function is constructed by combining the fusion features:

[0066] ;

[0067] in, Represent the noise objective function. express Follows a normal distribution. The symbol representing the normal distribution. This indicates that the real noise is sampled according to probability. This represents a discrete uniform distribution of real noise. Represents real noise. Indicates compliance with noise level Uniform distribution The symbol represents a uniform distribution; a mean square error is constructed based on all the predicted noise and all the true noise, and the conditional output features are inversely sampled based on the mean square error:

[0068] ;

[0069] ;

[0070] in, This represents the normal distribution that backsampling follows in the traditional diffusion model. Indicates by Samples obtained by inverse sampling , and Both represent noise scheduling coefficients. Represents variance. denoted as mean squared error; all samples with missing modes are replaced with the original samples obtained by inverse sampling to construct the training set.

[0071] In this process, for each diffusion step, a real sample (real noise) is defined, and a predicted sample (predicted noise) is generated through forward diffusion. The training objective is defined as minimizing the mean squared error between the predicted noise and the real noise, and a loss term (noise objective function) is generated by combining conditional and unconditional noise predictions. Then, in the recommendation phase, based on the given initial noise, multiple inverse diffusions are iteratively performed using a U-Net network structure with cross-attention, gradually restoring the initial noise to the original noise (original samples). These original samples are then used to fill in the corresponding parts of the training sentences with missing modalities, resulting in a complete training set covering all modalities. By capturing high-order semantic information from available modalities to guide the denoising process of the conditional diffusion model, the inherent complex uncertainties in MERC are decomposed, providing a comprehensive objective function for improving the performance of subsequent sentiment classification models. Furthermore, the constructed noise objective function can be used to optimize the conditional diffusion model, further enhancing its predictive performance.

[0072] Step S40: Calculate the feature uncertainty of the entire training set based on the mean square error, calculate the discrimination uncertainty of the entire training set based on the class probability simplex, and construct a fused probability vector based on the feature uncertainty and the discrimination uncertainty.

[0073] Existing evidence theories (DST, Dempster-Shafer Theory) typically represent the basic probability assignment (BPA) of the entire set based on empty uncertainty. However, empty uncertainty only reflects uncertainty from the perspective of "total amount of evidence," making it difficult to cover the complexity of all uncertainties in multimodal scenarios. Therefore, in the embodiments disclosed in this invention, DCEF (Dual Channel Evidence Fusion) is proposed, which characterizes complex uncertainties by simultaneously modeling uncertainty at both the feature source level and the discrimination level.

[0074] Specifically, the feature uncertainty of the entire training set is calculated based on the mean square error:

[0075] ;

[0076] in, Indicates the complete training set. Represents characteristic uncertainty. Indicates the first Class scaling factor Indicates the scaling factor. This represents the mean of the mean squared error. Mean square error, Represent the logistic function; construct the category probability simplex, and calculate the discrimination uncertainty of the entire training set based on the category probability simplex for each category:

[0077] ;

[0078] in, Indicates the uncertainty of the decision. Indicates the number of categories, Indicates the first The class probability simplex; based on the feature uncertainty and the discrimination uncertainty, the integration uncertainty of the entire training set is constructed:

[0079] ;

[0080] ;

[0081] in, Indicates integration uncertainty. Indicates the first The integration uncertainty of the class; the training set is divided into multiple subsets, and the integration uncertainty of all subsets is fused to obtain the probability vector of the training set:

[0082] ;

[0083] in, , and They represent subsets, Indicates a fusion operation. and They represent and Integration uncertainty, This represents the empty set.

[0084] Evidence theory correlates the parameters of the Dirichlet distribution with the belief distribution. Evidence theory states that a conventional neural network can be transformed into an "evidence neural network" simply by replacing the final softmax layer with a ReLU activation layer; in this case, the non-negative output of ReLU is defined as "evidence." In the embodiments disclosed in this invention, the dual-channel method for estimating the uncertainty of the entire training set is mainly illustrated using the single-modal case.

[0085] At the feature source level, the mean squared error (MSE) between "predicted noise and actual noise" during the conditional diffusion model sampling process is used to estimate the uncertainty of the recovered features, and the feature uncertainty of the entire set is given accordingly. Then, starting from the original available features, the feature uncertainty is initialized to 0, and at the discrimination level, cross-entropy is used to estimate the discrimination uncertainty (where the higher the entropy value, the more ambiguous the decision and the stronger the uncertainty). Finally, the two are iteratively fused to obtain the fused probability vector.

[0086] This invention models uncertainty simultaneously at both the feature source and discrimination levels and fuses it using Dirichlet-evidence (DST / Dirichlet-evidence): First, it uses the "Noise Matching Error (MSE)" in the conditional diffusion sampling process to measure the generation uncertainty of recovered features, mapping it to the uncertainty of the entire training set. Second, it uses the entropy / evidence strength of the discrimination output to characterize decision uncertainty, obtaining a second-channel uncertainty. Subsequently, it performs pairwise iterative synthesis using conflict management rules and introduces three uncertainty estimates to decompose three types of uncertainty sources: insufficient evidence, conflicting evidence, and consistent evidence, forming a dual-channel fusion paradigm distinct from existing single-channel confidence or posterior temperature calibration. When a modality recovery is unstable or the original modality is contaminated by noise, the high uncertainty at the source level automatically suppresses its evidence weight, avoiding misleading discrimination. Using Dirichlet-evidence instead of the Softmax posterior, combined with uncertainty estimation for constraints, makes the predicted confidence more consistent with the actual usable information, reducing overconfidence. Finally, the uncertainty-based synthesis, in cases of high conflict, returns the uncertainty quality to the entire set, reducing the amplification and propagation of erroneous evidence in the multimodal link.

[0087] Furthermore, the dual-channel evidence fusion method disclosed in the above embodiments can also be replaced by teacher-student distillation. During training, the full-modal teacher outputs soft labels and uncertainties, while the student fits the data under missing / recovered modal conditions. During inference, explicit evidence fusion is not required, and the student learns "when to believe whom" on their own. The evidence theory can also be replaced by various rules: Yager's Combination Rule distributes conflict quality across the entire set, which is conservative and lacks confidence; Dubois-Prade (a rule for handling the aggregation and inference of uncertain information) transfers conflicts to the union, suitable for partial class compatibility; PCR5 / 6 (proportional conflict redistribution) redistributes conflicts proportionally to the conflict sources, which is generally more stable.

[0088] Step S50: Determine multiple uncertainty estimates, construct a divergence regularization term, optimize the emotion classification network based on the noise objective function, all the uncertainty estimates, the divergence regularization term, and the probability vector, input the test set into the optimized emotion classification network, and output the emotion recognition result.

[0089] In optimizing the sentiment classification model, three types of uncertainty estimation are constructed to describe the situations of empty evidence, conflicting evidence, and consistent evidence. Consistent uncertainty estimation is used to assess cross-modal semantic confusion (caused by differences in evidence between different modalities). Reducing consistent uncertainty estimation can reduce uncertainty at the feature source level. Conflict uncertainty estimation characterizes evidence conflict within a modality. Reducing conflict uncertainty estimation can reduce uncertainty at the discrimination level. However, reducing conflict uncertainty estimation may lead to a decrease in the overall amount of evidence. Therefore, this invention uses empty uncertainty estimation as a constraint to alleviate the decrease in the overall amount of evidence.

[0090] Specifically, an evidence vector is constructed for each sample in the training set; corresponding Dirichlet parameters and Dirichlet intensities are constructed based on the evidence vectors; and a probability estimate of the class of each sample is constructed based on all the Dirichlet parameters and all the Dirichlet intensities.

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] in, Indicating in modality The next A vector of evidence, , and They represent Class 1, Class 2, and Class 3 respectively. Class , express Dirichlet parameters, Indicates the first Class Dirichlet parameters, express Dirichlet strength, Indicates the first Class of samples A probability estimate; based on all said categories and the Dirichlet intensity of each said category, construct an empty uncertainty estimate for each Dirichlet parameter:

[0096] ;

[0097] in, This indicates an estimation of uncertainty in open space. Represents the Dirichlet parameters. Indicates the number of categories, express The Dirichlet intensity; based on the probability estimates of the different categories, construct conflict uncertainty estimates for each Dirichlet parameter:

[0098] ;

[0099] in, express Conflict uncertainty estimation and The first Class and First Probability estimation of class samples; constructing consistent uncertainty estimates between any two samples based on the sample size of a single batch:

[0100] ;

[0101] in, This represents a consistent uncertainty estimate. Indicates the number of samples. Indicates the number of sample pairs. and They represent the first The first class of samples The and the first Dirichlet parameters for each sample, Let represent a constant; construct a divergence regularization term, and construct the final optimization objective function based on the noise objective function, the open space uncertainty estimate, the conflict uncertainty estimate, the uniform uncertainty estimate, and the divergence regularization term:

[0102] ;

[0103] ;

[0104] in, Denotes the divergence regularization term. Indicates the number of Dirichlet parameters. Indicates the first One Dirichlet parameter, express The estimated value, Indicates the Dirichlet distribution. Indicates a uniform Dirichlet distribution. express The probability, This indicates element-wise multiplication;

[0105] ;

[0106] in, This represents the final optimization objective function. express The noise objective function, express Unique hot-coded tags, express The divergence regularization term, , , and All of these represent hyperparameters. , and They represent The network is used to estimate the consistent uncertainty, conflict uncertainty, and empty space uncertainty. The constructed emotion classification network is then optimized using the final optimization objective function to obtain the target emotion classification model. The test set is then input into the target emotion classification model to output the emotion recognition result.

[0107] In order to enable the model to focus on improving evidence for the correct category and suppressing evidence for the wrong category, this invention adds a divergence regularization term to the objective function for training the model to punish false evidence and regularize the prediction distribution. At the same time, it combines all uncertainty estimates and uses hyperparameters (where each hyperparameter is determined by grid search) as constraints for each loss term to construct the final optimization objective function for training the sentiment classification network.

[0108] Furthermore, in order to verify the effectiveness of the target emotion classification model proposed in this invention, experiments were conducted on two benchmark datasets in the embodiments disclosed in this invention: IEMOCAP (Interactive Emotional Dyadic Motion Capture) and MELD (Multimodal Emotion Lines Dataset).

[0109] Based on these two datasets, a comprehensive comparative experiment was conducted using the HyperEF model proposed in this invention and other MERC methods. The models compared were all modality recovery models: MMIN (Missing Modality Imagination Network, the first method), GCNet (Global Context Network, the second method), IMDer (Incomplete Multimodality-Diffused Emotion Recognition, the third method), CIF-MMIN (Contrastive Learning-based Modality-Invariant Feature for Missing Modality Imagination Network, the fourth method), and SDR-GNN (Spectral Domain Reconstruction Graph Neural Network, the fifth method). The specific comparison results are shown in Table 1 below.

[0110] Table 1: Comparison Results

[0111]

[0112] The proposed HyperEF achieves optimal performance across all datasets and all missing data scenarios. Specifically, on IEMOCAP4, with a missing data rate of 0.1, HyperEF outperforms SDR-GNN by 3.5%. On the IEMOCAP6 benchmark, HyperEF's advantage becomes more pronounced as the missing data rate increases (a 4.0% improvement at a missing data rate of 0.6), primarily due to its MHGAT-based semantic feature reconstruction mechanism. Similarly, on the MELD dataset, HyperEF also achieves the best sentiment recognition performance.

[0113] Furthermore, in another embodiment disclosed in this invention, such as Figure 4 As shown, this demonstrates HyperEF's feature recovery capability relative to IMDer. In this embodiment, the recovered features are reduced to two dimensions along with the original features, and visualized using t-SNE (t-Distributed Stochastic Neighbor Embedding). The results show that HyperEF outperforms IMDer on both datasets. In IEMOCAP (i.e., ... Figure 4 In the first dataset (MELD), the recovered features across the three modalities are highly similar to the distributions of the original features; in MELD (i.e., Figure 4 In the second dataset of MELD, the recovery quality of the visual and audio modalities is also high, but the recovery distribution of the text modalities deviates from the original distribution. This is because it is difficult to directly extract semantic information from the visual and audio modalities of MELD, resulting in a lack of corresponding semantic conditions when recovering the text modalities, thus causing a shift in the recovery distribution.

[0114] This invention achieves adaptive evidence fusion by simultaneously estimating uncertainty at both the feature source and discrimination levels through dual-channel evidence fusion. This mitigates the performance degradation caused by modality loss, explicitly recovers the latent features of missing modalities in the feature space, and improves the accuracy of the final emotion recognition.

[0115] Furthermore, such as Figure 5 As shown, based on the above-mentioned multimodal emotion recognition method based on hypergraph diffusion and evidence fusion, this invention also provides a multimodal emotion recognition system based on hypergraph diffusion and evidence fusion, wherein the multimodal emotion recognition system based on hypergraph diffusion and evidence fusion includes:

[0116] The masked hypergraph module 51 is used to acquire multiple training statements and multiple modalities of each training statement, construct a hypergraph network using mask vectors, and aggregate the node set and hyperedge set in the hypergraph network to update all nodes in the hypergraph network and obtain the corresponding node representations.

[0117] Feature fusion module 52 is used to concatenate all the node representations according to the modal dimension to obtain conditional information, input the conditional information into the conditional diffusion model to obtain multiple output features, and construct fused features based on all the output features;

[0118] The inverse sampling module 53 is used to perform inverse sampling based on the condition information to fill in all missing modes and generate mean squared error and training set;

[0119] The dual-channel evidence fusion module 54 is used to calculate the feature uncertainty of the entire training set based on the mean square error, calculate the discrimination uncertainty of the entire training set based on the class probability simplex, and construct a fused probability vector based on the feature uncertainty and the discrimination uncertainty.

[0120] The model training module 55 is used to determine multiple uncertainty estimates, construct a divergence regularization term, optimize the emotion classification network based on all the uncertainty estimates, the divergence regularization term and the probability vector, input the test set into the optimized emotion classification network, and output the emotion recognition result.

[0121] Furthermore, such as Figure 6 As shown, based on the above-mentioned multimodal emotion recognition method and system based on hypergraph diffusion and evidence fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0122] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard drive or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multimodal emotion recognition program 40 based on hypergraph diffusion and evidence fusion, which can be executed by the processor 10 to implement the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion in this application.

[0123] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion.

[0124] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0125] In one embodiment, when the processor 10 executes the multimodal emotion recognition program 40 based on hypergraph diffusion and evidence fusion in the memory 20, it implements the steps of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion as described above.

[0126] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal emotion recognition program based on hypergraph diffusion and evidence fusion, and the multimodal emotion recognition program based on hypergraph diffusion and evidence fusion, when executed by a processor, implements the steps of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion as described above.

[0127] In summary, this invention provides a multimodal emotion recognition method and related device based on hypergraph diffusion and evidence fusion. The method includes: acquiring multiple training statements and multiple modalities of each training statement; constructing a hypergraph network using mask vectors; aggregating the node set and hyperedge set in the hypergraph network to update all nodes in the hypergraph network and obtain corresponding node representations; concatenating all node representations according to modality dimensions to obtain conditional information; inputting the conditional information into a conditional diffusion model to obtain multiple output features; constructing fused features based on all output features; performing inverse sampling based on the conditional information to fill in all missing modalities and generating mean squared error and a training set; calculating the feature uncertainty of the training set based on the mean squared error; calculating the discrimination uncertainty of the training set based on the class probability simplex; constructing a fused probability vector based on the feature uncertainty and the discrimination uncertainty; determining multiple uncertainty estimates; constructing a divergence regularization term; optimizing the emotion classification network based on all uncertainty estimates, the divergence regularization term, and the probability vector; inputting the test set into the optimized emotion classification network; and outputting emotion recognition results. This invention achieves adaptive evidence fusion by simultaneously estimating uncertainty at both the feature source and discrimination levels through dual-channel evidence fusion. This mitigates the performance degradation caused by modality loss, explicitly recovers the latent features of missing modalities in the feature space, and improves the accuracy of the final emotion recognition.

[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0129] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0130] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal emotion recognition method based on hypergraph diffusion and evidence fusion, characterized in that, The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion includes: Multiple training statements and multiple modalities of each training statement are obtained. A hypergraph network is constructed using mask vectors. The set of nodes and the set of hyperedges in the hypergraph network are aggregated to update all nodes in the hypergraph network and obtain the corresponding node representations. All node representations are concatenated according to modal dimension to obtain conditional information. The conditional information is input into the conditional diffusion model to obtain multiple output features. A fusion feature is constructed based on all the output features. Inverse sampling is performed based on the conditional information to fill in all missing modes and generate mean squared error and training set; Based on the mean squared error, calculate the feature uncertainty of the entire training set; based on the class probability simplex, calculate the discrimination uncertainty of the entire training set; and construct a fused probability vector based on the feature uncertainty and the discrimination uncertainty. Multiple uncertainty estimates are determined, a divergence regularization term is constructed, and the emotion classification network is optimized based on all the uncertainty estimates, the divergence regularization term, and the probability vector. The test set is then input into the optimized emotion classification network, and the emotion recognition result is output, specifically including: The softmax layer is replaced with a ReLU activation layer to construct an evidence vector for each sample in the entire training set based on the probability vector. The corresponding Dirichlet parameters and Dirichlet intensities are then constructed based on the evidence vectors. Finally, a probability estimate of the class for each sample is constructed based on all the Dirichlet parameters and all the Dirichlet intensities. ； ； ； ； in, Indicating in modality The next A vector of evidence, , and They represent Class 1, Class 2, and Class 3 respectively. Class , express Dirichlet parameters, Indicates the first Class Dirichlet parameters, express Dirichlet strength, Indicates the first Class of samples A probability estimate; Based on all the categories and the Dirichlet intensity for each category, construct an estimate of the empty uncertainty for each Dirichlet parameter: ； in, This indicates an estimation of uncertainty in open space. Represents the Dirichlet parameters. Indicates the number of categories, express Dirichlet strength; Based on the probability estimates for the different categories, construct conflict uncertainty estimates for each Dirichlet parameter: ； in, express Conflict uncertainty estimation and The first Class and First Probability estimation of class samples; Based on the sample size of a single batch, construct a consistent uncertainty estimate between any two samples: ； in, This represents a consistent uncertainty estimate. Indicates the number of samples. Indicates the number of sample pairs. and They represent the first The first class of samples The and the first Dirichlet parameters for each sample, Represents a constant; Construct a divergence regularization term and a cross-entropy loss function, and build the final optimization objective function based on the empty space uncertainty estimate, the conflict uncertainty estimate, the consistent uncertainty estimate, and the divergence regularization term: ； ； in, Denotes the divergence regularization term. Indicates the number of Dirichlet parameters. Indicates the first One Dirichlet parameter, express The estimated value, Indicates the Dirichlet distribution. Indicates a uniform Dirichlet distribution. express The probability, This indicates element-wise multiplication; ； in, This represents the final optimization objective function. express The cross-entropy loss function, express Unique hot-coded tags, express The divergence regularization term, , , and All of these represent hyperparameters. , and They represent Consistent uncertainty estimation, conflict uncertainty estimation, and open space uncertainty estimation; The constructed emotion classification network is optimized using the final optimization objective function to obtain the target emotion classification model. The test set is then input into the target emotion classification model to output the emotion recognition result.

2. The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion according to claim 1, characterized in that, The step of acquiring multiple training statements and multiple modalities for each training statement, and constructing a hypergraph network using mask vectors, specifically includes: Obtain training dialogues, and construct multiple training statements and multiple modalities for each training statement based on the training dialogues: ； in, Indicates training dialogue, , and They represent the 1st, 2nd, and 3rd respectively. One training statement, , and They represent , and The statement object; ； in, Indicates the first One training statement, , and They represent The text modality, visual modality, and audio modality; Define a corresponding mask for each of the aforementioned modalities, randomly define multiple of the aforementioned modalities as missing modalities, and embed the corresponding masks into the positions corresponding to all the missing modalities to construct a hypergraph network: ； in, This represents a hypergraph network. , , and They represent , , and The mask, , and They represent The masks corresponding to the text modality, visual modality, and audio modality; ； ； in, This represents the set of nodes in a hypergraph network. and They represent the 1st and the 2nd respectively. 1 node Denotes the set of superedges in a hypergraph network. , and These represent the 1st, 2nd, and 3rd context edges, respectively. and They represent the 1st and the 2nd respectively. A multimodal edge; ； in, This indicates an embedding operation. Indicates the first A node after embedding a mask. Indicates the first Each node.

3. The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion according to claim 2, characterized in that, The aggregation of the node set and hyperedge set in the hypergraph network to update all nodes in the hypergraph network and obtain the corresponding node representation specifically includes: Construct a hyperedge attention score. For each hyperedge, assign weights to each node on the hyperedge based on the hyperedge attention score, and then aggregate each node on the hyperedge to update the hyperedge representation, until all hyperedges have been updated: ； in, Indicates the first The updated hyperedge representation after layer update Represents a non-linear activation function. Indicates the first superedge and the first The super-edge attention score between nodes Indicates the first The set of nodes for an edge. Indicates the first The first layer Each node represents a node. Indicates weight; ； in, Represents an exponential function. This represents the activation function. This represents a trainable weight vector. Indicates weight, This indicates a splicing operation. Indicates the first The updated hyperedge representation after layer update Indicates the first The first layer Each node represents a node. Indicates the first 1 node Indicates the first The first layer Each node represents a node; Construct node attention scores. For each node, assign weights to all updated hyperedges connected to that node based on the node attention score. Then, aggregate all hyperedges to update the node, obtaining the node representation of that node. Repeat this process until all nodes have been updated. ； in, Indicates the first The updated node representation of the layer. Indicates the first A super edge, Indicates and The set of connected hyperedges Indicates the first The node and the first Attention scores between nodes along the superedges; ； in, This represents a trainable weight vector. Indicates the first The first layer Each hyperedge is represented.

4. The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion according to claim 1, characterized in that, The process involves concatenating all node representations along the modal dimension to obtain conditional information, inputting this conditional information into a conditional diffusion model to obtain multiple output features, and constructing a fusion feature based on all the output features. Specifically, this includes: Based on the dimension of the modality represented by each node, all the node representations are concatenated to obtain the condition information; The conditional information is input into the conditional diffusion model, which performs cross-attention fusion between the input features and the conditional information to obtain multiple output features. Conditional information is randomly filtered out from multiple output features with a fixed probability to obtain multiple unconditional output features. All unconditional output features and the unfiltered output features are then weighted and fused to obtain a fused feature. ； in, Indicates the first The fusion characteristics of step diffusion, The weights represent the feature fusion weights. This indicates conditional output features. Indicates unconditional output features. Indicates the first Step-diffusion samples, Indicates the number of diffusion steps. Indicates conditional information.

5. The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion according to claim 4, characterized in that, The step of performing inverse sampling based on the conditional information to fill in all missing modes and generate mean squared error and the complete training set specifically includes: Extract multiple real noises and multiple predicted noises generated during forward diffusion of the conditional diffusion model; A mean squared error is constructed based on all the predicted noise and all the true noise, and the conditional output features are inversely sampled based on the mean squared error: ； ； in, This represents the normal distribution that backsampling follows in the traditional diffusion model. Indicates by Samples obtained by inverse sampling , and Both represent noise scheduling coefficients. Represents variance. Indicates mean square error; All samples with missing modalities are replaced with the original samples obtained by inverse sampling to construct the training set.

6. The multimodal emotion recognition method based on hypergraph diffusion and evidence fusion according to claim 5, characterized in that, The step of calculating the feature uncertainty of the entire training set based on the mean square error, calculating the discrimination uncertainty of the entire training set based on the class probability simplex, and constructing a fused probability vector based on the feature uncertainty and the discrimination uncertainty specifically includes: Based on the mean square error, calculate the feature uncertainty of the entire training set: ； in, Indicates the complete training set. Represents characteristic uncertainty. Indicates the first Class scaling factor Indicates the scaling factor. This represents the mean of the mean squared error. Mean square error, Represents a logical function; Construct a class probability simplex, and calculate the discrimination uncertainty of the entire training set based on the class probability simplex for each class: ； in, Indicates the uncertainty of the decision. Indicates the number of categories, Indicates the first The simplex of class probabilities; Based on the feature uncertainty and the discrimination uncertainty, construct the integrated uncertainty of the entire training set: ； ； in, Indicates integration uncertainty. Indicates the first Integration uncertainty of classes; The entire training set is divided into multiple subsets, and the integration uncertainty of all subsets is fused to obtain the probability vector of the entire training set: ； in, , and They represent subsets, Indicates a fusion operation. and They represent and Integration uncertainty, This represents the empty set.

7. A multimodal emotion recognition system based on hypergraph diffusion and evidence fusion, characterized in that, The multimodal emotion recognition system based on hypergraph diffusion and evidence fusion is used to implement the multimodal emotion recognition system method based on hypergraph diffusion and evidence fusion as described in any one of claims 1-6, including: The masked hypergraph module is used to acquire multiple training statements and multiple modalities of each training statement, construct a hypergraph network using mask vectors, and aggregate the node set and hyperedge set in the hypergraph network to update all nodes in the hypergraph network and obtain the corresponding node representations. The feature fusion module is used to concatenate all the node representations according to the modal dimension to obtain conditional information, input the conditional information into the conditional diffusion model to obtain multiple output features, and construct a fused feature based on all the output features; The inverse sampling module is used to perform inverse sampling based on the condition information to fill in all missing modes and generate mean squared error and training set; The dual-channel evidence fusion module is used to calculate the feature uncertainty of the entire training set based on the mean square error, calculate the discrimination uncertainty of the entire training set based on the class probability simplex, and construct a fused probability vector based on the feature uncertainty and the discrimination uncertainty. The model training module is used to determine multiple uncertainty estimates, construct a divergence regularization term, optimize the emotion classification network based on all the uncertainty estimates, the divergence regularization term, and the probability vector, and input the test set into the optimized emotion classification network to output the emotion recognition result.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multimodal emotion recognition program based on hypergraph diffusion and evidence fusion stored in the memory and executable on the processor. When the multimodal emotion recognition program based on hypergraph diffusion and evidence fusion is executed by the processor, it implements the steps of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal emotion recognition program based on hypergraph diffusion and evidence fusion, which, when executed by a processor, implements the steps of the multimodal emotion recognition method based on hypergraph diffusion and evidence fusion as described in any one of claims 1-6.

Images