A multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization

Abstract

A multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization is proposed. The method acquires a multimodal dialogue dataset; constructs a multimodal feature extraction module for encoding acoustic, visual, and textual features; constructs a graph structure module for capturing long-distance intramodal dependencies through intramodal edge propagation and for realizing feature interaction through intermodal edges; constructs a multimodal dynamic fusion module combining improved graph convolutional layers and dynamic gating mechanisms; constructs a multimodal mutual information maximization module for estimating the lower bound of mutual information and generating the mutual information loss function of the fusion layer; incorporates an emotion classifier to form the initial recognition model; trains and optimizes the model; and performs online emotion recognition. This method is simple to implement, low in cost, and highly accurate. By modeling cross-modal dynamic interaction fusion through graph structure modeling and enhancing feature complementarity through mutual information constraints, it significantly improves the accuracy and robustness of dialogue emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of intelligent emotion recognition technology, specifically involving a multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization. Background Technology

[0002] In recent years, with the rapid development of intelligent interactive systems (such as virtual assistants, social robots, and remote psychological counseling platforms), emotion recognition technology has become a core driving force for achieving natural and empathetic human-computer interaction. Early research focused on unimodal emotion analysis, such as text-based emotion classification and voice emotion analysis. However, these methods cannot fully capture the complexity of human emotions and often misjudge true emotions by ignoring the synergistic effects and inconsistencies of cross-modal signals. For example, a user may deny negative emotions through language in a conversation, but convey their true emotional state through a low tone of voice, evasive eye contact, or stiff body language. The inconsistency and time dependence of such multimodal signals significantly limit unimodal analysis methods, thus researchers have gradually shifted their attention to multimodal fusion.

[0003] Multimodal emotion recognition (MER) has made significant progress in the accuracy and robustness of emotion classification by fusing multimodal data such as text, speech, and vision. Thanks to the construction of large-scale labeled datasets and advancements in deep learning techniques, models can capture cross-modal associations in a more refined manner. To accurately identify complex emotions, multimodal methods significantly improve classification reliability by addressing inconsistencies between cross-modal signals. For example, MultiEMO designs a relevance-aware multimodal fusion framework that uses hierarchical gating units to dynamically quantify the mutual information entropy of text-speech modal pairs, effectively distinguishing between genuine and feigned anger. Meanwhile, Graph Neural Networks (MGNNs) combine multi-task learning with discrete classification and dimensionality regression in an "exploration-exploitation" approach to model the potential conflict between text sentiment polarity and speech arousal. These techniques significantly improve model accuracy by quantifying intermodal adversarial features and transforming cross-modal conflicts into interpretable decision-making criteria.

[0004] Furthermore, through techniques such as contrastive learning and cross-modal alignment, models can uncover potential semantic consistency among text, speech, and vision. For example, in cross-modal semantic alignment, contrastive learning methods introduced in transferable visual models construct a language-driven cross-modal semantic space by supervising visual representation learning through text descriptions. Motion-anchored contrastive learning frameworks capture contextual dependencies in dialogues using temporal sentiment anchors. At the algorithmic level, the cross-modal representation learning capability of deep learning becomes a core driving force. Transformer-based models (such as MulT) dynamically align the time steps of different modalities through self-attention mechanisms, solving the asynchrony problem between text, speech, and visual signals. Further research in this field proposes a modality-adaptive context-aware framework for adaptive context and modal interaction modeling, introducing a gating mechanism to dynamically adjust the contribution weights of different modalities.

[0005] However, most existing methods primarily use simple concatenation or weighted summation for cross-modal sentiment representation fusion. This superficial fusion approach often introduces redundancy into features with overlapping semantic information, while amplifying noise interference from low-quality modalities. Furthermore, they have limited capabilities in modeling dynamic interactions and long-range contextual dependencies, failing to model dynamic contextual understanding over time series, and are insufficient for cross-modal interaction reasoning.

[0006] In summary, existing multimodal emotion recognition technologies have significant shortcomings in areas such as deep cross-modal dependency mining, dynamic interaction modeling, and long-range context capture. There is an urgent need to provide an innovative method that can dynamically adjust information flow, reduce redundant interference, and systematically model long-range cross-modal dependencies to improve emotion recognition performance in complex dialogue scenarios. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization. This method is simple to implement, has low implementation cost, and high recognition accuracy. By modeling cross-modal dynamic interaction fusion through graph structure modeling and enhancing feature complementarity by utilizing mutual information constraints, it can significantly improve the accuracy and robustness of dialogue emotion recognition.

[0008] To achieve the above objectives, this invention provides a multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization, comprising the following steps:

[0009] Step 1: Obtain the multimodal dialogue dataset;

[0010] Step 2: Construct a multimodal feature extraction module to extract multimodal features;

[0011] Step 3: Construct the graph structure module; construct an undirected graph based on the number of utterances in the dialogue, with each utterance represented by three nodes corresponding to the acoustic, visual, and text modalities; initialize the node embeddings in the graph, and construct intra-modal and inter-modal edges according to the edge connection rules to form a dialogue-based graph structure module; the graph structure module is used to calculate the edge weights. Simultaneously, it is used to propagate features through intramodal edges, capture long-distance intramodal dependencies, realize feature interaction through intermodal edges, and output the final feature representation of multimodal nodes. ;

[0012] Step 4: Construct a multimodal dynamic fusion module to fuse multimodal features;

[0013] Step 5: Construct a multimodal mutual information maximization module; Based on the Barber-Agakov lower bound estimation method, construct a mutual information maximization module including a mean prediction network and a variance prediction network. This module is used to estimate the lower bound of mutual information, based on the features between the three modalities and the features of a single modality. With fusion modal features Maximizing mutual information between them generates the mutual information loss function of the fusion layer. ;

[0014] Step Six: Construct a multimodal emotion recognition model; S61: Based on the multimodal feature extraction module, graph structure module, multimodal dynamic fusion module, and multimodal mutual information maximization module, add an emotion classifier to form the initial recognition model; simultaneously, construct a model containing a mutual information loss function for the fusion layer. Total loss function S62: Train the initial recognition model using the training and test sets, and use the total loss function during training. The model parameters were optimized, and a multimodal emotion recognition model was obtained after training.

[0015] Step 7: Perform online emotion recognition; collect a sentence containing original acoustic features. Original visual features and original text features The dialogue data is used as prediction input data; the prediction input data is then fed into a multimodal emotion recognition model for recognition, and emotion labels are output.

[0016] As a preferred option, the process of obtaining the multimodal dialogue dataset in step one is as follows: Extracting the original acoustic features from the dialogue dataset containing acoustic, visual, and textual information. Original visual features and original text features and ensure the original acoustic characteristics Original visual features and original text features The timestamps are aligned to form a sample dataset; the sample dataset is then divided into a training set and a test set according to a set ratio.

[0017] As a preferred option, the process of constructing the multimodal feature extraction module in step two is as follows:

[0018] A multimodal feature extraction module was built based on OpenSmile, Roberta, and DenseNet; OpenSmile, Roberta, and DenseNet were used to extract the original acoustic features. Original visual features and original text features Encoding is performed, and speech features are obtained according to formulas (1), (2), and (3) respectively. Visual features and text features ;

[0019] (1);

[0020] (2);

[0021] (3);

[0022] In the formula, These are parameters for the OpenSmile tool. These are the parameters of the DenseNet model. These are the parameters for the Roberta model.

[0023] Furthermore, in order to better capture long-distance dependencies within a modality and to better facilitate cross-modal information interaction, in step three, when constructing the undirected graph, each utterance is represented by three nodes corresponding to the acoustic, visual, and textual modalities. In addition, the weights of the edges are calculated based on the cosine similarity function according to formula (4). ;

[0024] (4);

[0025] In the formula, For nodes The embedding vector; For nodes The embedding vector; Cosine similarity; This is used to convert similarity into angles and normalize them to the [0,1] interval.

[0026] Furthermore, in order to effectively aggregate intramodal and intermodal contextual information, the process of constructing the multimodal dynamic fusion module in step four is as follows:

[0027] A multimodal dynamic fusion module is constructed based on an improved graph convolutional layer and by introducing a dynamic gating mechanism. The multimodal dynamic fusion module has a multi-layer structure, with the final feature representation of multimodal nodes. As input, each layer first performs intra-modal feature aggregation through residual connections and dynamic weights to obtain the graph convolution output. As shown in formula (5), enhanced output is obtained by dynamically fusing cross-modal contextual information through a gating mechanism based on feature aggregation. As shown in formula (6), finally, through the stacking of multi-layer structures, the multimodal contextual features are fused into the dynamic semantic space, and the fused modal features of each sentence are output. ;

[0028] (5);

[0029] In the formula, For the multimodal dynamic fusion module, the first Graph convolution output of the layer; It is a non-linear activation function; The residual coefficient; For the first The residual state after layer graph convolution; It is a renormalized graph convolution matrix. , For degree matrix, It is an adjacency matrix; For the initial multimodal features, by , , Initialization yielded; This is the inter-layer gating coefficient. , For hyperparameters; The weight matrix is ​​a learnable weight matrix; The identity mapping matrix;

[0030] (6);

[0031] In the formula, For the multimodal dynamic fusion module, the first Enhanced output of the layer; This is a gating signal;

[0032] Furthermore, in order to fully learn the inherent ordering patterns of contextual information in different semantic spaces, control the flow of information between layers, reduce redundant information, and enhance the complementarity between modalities, the process of dynamically fusing cross-modal contextual information through a gating mechanism in step four is as follows:

[0033] S41: Calculate the gate signal according to formula (7) , obtained the Layer nodes Aggregation characteristics;

[0034] (7);

[0035] In the formula, It is the sigmoid function; This is the gate weight matrix; For the first Gated outputs of the layer; For the first Enhanced hidden states of the layer; This is a gated bias term;

[0036] S42: Generate candidate memory units according to formula (8) To store the first Context information of the layer;

[0037] (8);

[0038] In the formula, This is the weight matrix of the memory units; For memory cell bias terms;

[0039] S43: By updating the gate And the Gate of Oblivion Update the information above and below, and obtain the updated memory unit according to formula (9). ;

[0040] (9);

[0041] S44: Through the output gate To selectively output information; This indicates element-wise multiplication.

[0042] In order to effectively filter out modality-specific noise that is irrelevant to the task, in step five, when the mutual information maximization module is used to estimate the lower bound of mutual information, it obtains the mutual information according to formula (10). ;

[0043] (10);

[0044] In the formula, It is a multivariate Gaussian distribution. , Indicates by parameters The mean vector predicted by the controlled neural network. Indicates by parameters The variance scalar of the controlled neural network predictions. The fact that the covariance matrix is ​​the identity matrix indicates that the covariance matrix of this Gaussian distribution is a diagonal matrix and that the variances of each dimension are equal. KL divergence is used to measure the true conditional distribution. and multivariate Gaussian distribution Differences; It is a variational lower bound estimate of mutual information; This is a constant term used to ensure that the lower bound is consistent with the direction of mutual information.

[0045] Furthermore, to preserve cross-modal invariance and ensure the accuracy of the trained model, in step five, the model is evaluated based on the features across the three modalities and the features within a single modality. With fusion modal features Maximizing mutual information between them generates the mutual information loss function of the fusion layer. The process is as follows:

[0046] S51: Prediction results of fused modal features Z based on formula (11) Perform L2 normalization and obtain the unit vector. ;

[0047] (11);

[0048] In the formula, It is a parameter neural networks, It is the Euclidean norm, which is used to normalize a vector to obtain a vector of unit length.

[0049] S52: Based on formula (12), single-modal features vectors in Perform L2 normalization and obtain the vector. normalized value ;

[0050] (12);

[0051] S53: Calculate the normalized value using the dot product. and The similarity is obtained by formula (13) to obtain the modal feature similarity score. ;

[0052] (13);

[0053] S54: Obtain the joint loss function according to formula (14) ;

[0054] (14);

[0055] In the formula, for and Similarity score; For all single-modal features Mid-vector With fusion modal features The sum of similarities is used to normalize the score; The desired operation is denoted as for single-modal features. Mid-vector The statistical average;

[0056] S55: Obtain the mutual information loss function of the fusion layer according to formula (15) ;

[0057] (15);

[0058] In the formula, Representing visual modalities The mutual information loss with the fused modal features Z, Representing acoustic modes The mutual information loss with the fused modal features Z, Representing text modality Mutual information loss with fused modal features Z.

[0059] Furthermore, in order to effectively optimize the model parameters, in step S61 of step six, the total loss function is obtained according to formula (16). ;

[0060] (16);

[0061] In the formula, Let cross-entropy be the loss function. For hyperparameters related to mutual information.

[0062] To improve the accuracy of emotion recognition, the initial recognition model is trained in step S62 as follows:

[0063] S62-1: Set the maximum number of training epochs, convergence threshold, and other hyperparameters required for training;

[0064] S62-2: Begin end-to-end training of the initial recognition model using the training and test sets; align the raw acoustic features with timestamps from a single dialogue sentence. Original visual features and original text features As input data, it is fed into the initial recognition model to predict and output the sentiment label in the dialogue.

[0065] In each round of training, the following operations are performed: determine whether the learning rate needs to be adjusted in the current round, pass the input data into the initial recognition model, predict and generate sentiment labels, calculate the error between the predicted sentiment labels and the real labels, calculate the loss function and backpropagate to update the parameters of each layer, calculate the difference between the loss function in the current round and the previous round, and compare the difference with the convergence threshold to determine whether convergence has occurred. When convergence occurs, the model parameters in the training process are saved first, and then the next round of training is performed. When convergence fails, the model parameters in the training process are not saved, and the next round of training is performed directly.

[0066] S62-3: After the loss function converges and the best result is obtained or the maximum number of training rounds is reached, the model training is completed. The model parameters after training are saved to obtain the multimodal emotion recognition model.

[0067] To address the shortcomings of existing multimodal emotion recognition technologies, such as insufficient deep cross-modal dependency mining, poor dynamic interaction modeling, and incomplete long-range context capture, this invention captures the constantly evolving interaction and dependency relationships in the context through dynamic graph networks, and optimizes multimodal fusion by employing cross-modal mutual information maximization and single-modal mutual information maximization strategies. Specifically, this invention proposes a multimodal dialogue emotion recognition method based on graph dynamic fusion and mutual information maximization. First, dialogue data containing acoustic, visual, and textual information is collected, and the original features of each modality are extracted separately. Then, a sample dataset is constructed based on timestamp alignment, providing a reliable data foundation for the subsequent training of the recognition model. Second, a multimodal feature extraction module is constructed using OpenSmile, Roberta, and DenseNet, which can be easily used to encode speech, vision, and text separately, thus efficiently obtaining speech, visual, and textual features. Next, each utterance is represented as a cross-modal node, and a dialogue-based graph structure module is constructed. This module can comprehensively capture multimodal information in the dialogue, avoiding the limitations of a single modality, and effectively promoting cross-modal feature interaction and enhancing feature complementarity. Subsequently, a multimodal dynamic fusion module is constructed based on an improved graph convolutional layer and a dynamic gating mechanism. This module can dynamically aggregate contextual information inside and outside the modality, and achieve deep fusion of multimodal features in the dynamic semantic space through hierarchical stacking, outputting the fused modal features of each utterance. Furthermore, based on Barber... The Agakov lower bound estimation method constructs a multimodal mutual information maximization module, which can implement mutual information maximization constraints between modalities and between single modalities and fused features, and generates a mutual information loss function for the fusion layer, enabling optimization of model parameters through end-to-end training. Finally, a sentiment classifier is introduced, which can efficiently and accurately predict the sentiment label of each dialogue through the calculation process of the Softmax function.

[0068] This method is simple to implement, low in cost, and highly accurate. It effectively reduces redundant information and enhances cross-modal complementarity by using graph structure modeling for cross-modal dynamic interaction fusion and mutual information constraints to enhance feature complementarity, thus significantly improving the accuracy and robustness of dialogue emotion recognition. Attached Figure Description

[0069] Figure 1 This is a flowchart of the present invention;

[0070] Figure 2 This is a schematic diagram of the confusion matrix of the multimodal emotion recognition model in this invention on the IEMOCAP dataset. Detailed Implementation

[0071] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0072] like Figure 1 and Figure 2 As shown, this invention provides a multimodal emotion recognition method based on graph dynamic fusion and mutual information maximization, comprising the following steps:

[0073] Step 1: Obtain the multimodal dialogue dataset;

[0074] Raw acoustic features were extracted from the dialogue dataset containing acoustic, visual, and textual information. Original visual features and original text features and ensure the original acoustic characteristics Original visual features and original text features The timestamps are aligned to form a sample dataset; the sample dataset is divided into a training set and a test set according to a set ratio; specifically, the original acoustic features are extracted from the speech in the dialogue dataset. Extracting raw visual features from video footage in the dialogue dataset Extracting text features from the text content of the dialogue dataset .

[0075] Step 2: Construct a multimodal feature extraction module to extract multimodal features;

[0076] A multimodal feature extraction module was built based on OpenSmile, Roberta, and DenseNet; OpenSmile, Roberta, and DenseNet were used to extract the original acoustic features. Original visual features and original text features Encoding is performed, and speech features are obtained according to formulas (1), (2), and (3) respectively. Visual features and text features ;

[0077] (1);

[0078] (2);

[0079] (3);

[0080] In the formula, These are parameters for the OpenSmile tool. These are the parameters of the DenseNet model. These are the parameters for the Roberta model.

[0081] Step 3: Construct the graph structure module;

[0082] An undirected graph is constructed based on the number of utterances in the dialogue. Each utterance is represented by three nodes corresponding to the acoustic, visual, and textual modalities. The total number of nodes in the graph is 3N, where N is the number of utterances. Node embeddings in the graph are initialized, and intra-modal and inter-modal edges are constructed according to edge connection rules to form a dialogue-based graph structure module. The graph structure module is used to calculate the edge weights. Simultaneously, it is used to propagate features through intramodal edges, capture long-distance intramodal dependencies, realize feature interaction through intermodal edges, and output the final feature representation of multimodal nodes. Multimodal node final feature representation Including text features Speech features Video features This ensures that the final feature representation of the output multimodal nodes retains the uniqueness of each modality, facilitating on-demand selection or fusion by downstream tasks;

[0083] In this way, we can not only effectively capture long-distance dependencies within a modality using graph structure modules, but also better conduct cross-modal information interaction.

[0084] To better capture long-distance dependencies within a modality and to facilitate cross-modal information interaction, when constructing the undirected graph, each utterance is represented by three nodes corresponding to the acoustic, visual, and textual modalities. Furthermore, the weights of the edges are calculated using the cosine similarity function according to formula (4). ;

[0085] (4);

[0086] In the formula, For nodes The embedding vector; For nodes The embedding vector; Cosine similarity; This is used to convert similarity into angles and normalize them to the [0,1] interval.

[0087] Step 4: Construct a multimodal dynamic fusion module to fuse multimodal features;

[0088] A multimodal dynamic fusion module is constructed based on an improved graph convolutional layer and by introducing a dynamic gating mechanism. The multimodal dynamic fusion module has a multi-layer structure, with the final feature representation of multimodal nodes. As input, each layer first performs intra-modal feature aggregation through residual connections and dynamic weights to obtain the graph convolution output. As shown in formula (5), enhanced output is obtained by dynamically fusing cross-modal contextual information through gating mechanisms (update gate, forget gate, and output gate) based on feature aggregation. As shown in formula (6), this effectively aggregates intramodal and intermodal contextual information. Finally, through the stacking of multi-layer structures, the multimodal contextual features are fused into the dynamic semantic space, and the fused modal features of each sentence are output. ;

[0089] (5);

[0090] In the formula, For the multimodal dynamic fusion module, the first Graph convolution output of the layer; It is a non-linear activation function; The residual coefficient; For the first The residual state after layer graph convolution; It is a renormalized graph convolution matrix. , For degree matrix, It is an adjacency matrix; For the initial multimodal features, by , , Initialization yielded; This is the inter-layer gating coefficient. , For hyperparameters; The weight matrix is ​​a learnable weight matrix; The identity mapping matrix;

[0091] (6);

[0092] In the formula, For the multimodal dynamic fusion module, the first Enhanced output of the layer; This is a gating signal.

[0093] To fully learn the inherent ordering patterns of contextual information in different semantic spaces, control the flow of information between layers, reduce redundant information, and enhance the complementarity between modalities, the process of dynamically fusing cross-modal contextual information through gating mechanisms (including update gates, forget gates, and output gates) is as follows:

[0094] S41: Calculate the gate signal according to formula (7) , obtained the Layer nodes Aggregation characteristics;

[0095] (7);

[0096] In the formula, It is the sigmoid function; This is the gate weight matrix; For the first Gated outputs of the layer; For the first Enhanced hidden states of the layer; This is a gated bias term;

[0097] S42: Generate candidate memory units according to formula (8) To store the first Context information of the layer;

[0098] (8);

[0099] In the formula, This is the weight matrix of the memory units; For memory cell bias terms;

[0100] S43: By updating the gate And the Gate of Oblivion Update the information above and below, update the door. Information used to control writing to memory; forget gate Used to remove redundant information and obtain the updated memory cell according to formula (9). ;

[0101] (9);

[0102] S44: Through the output gate To selectively output information; This indicates element-wise multiplication.

[0103] Step 5: Construct a multimodal mutual information maximization module;

[0104] A mutual information maximization module, comprising a mean prediction network and a variance prediction network, is constructed based on the Barber-Agakov lower bound estimation method. This module is used for mutual information lower bound estimation, based on the relationships between features of the three modalities and single-modal features. With fusion modal features Maximizing mutual information between them generates the mutual information loss function of the fusion layer. , to be used for training emotion recognition models;

[0105] By optimizing the mutual information edge estimation using a computable lower bound of mutual information (MI), task-irrelevant modality-specific noise can be effectively filtered out, while preserving cross-modality invariant content as much as possible.

[0106] To effectively filter out modality-specific noise irrelevant to the task, the mutual information maximization module obtains the mutual information according to formula (10) when estimating the lower bound of mutual information. ;

[0107] (10);

[0108] In the formula, It is a multivariate Gaussian distribution. , Indicates by parameters The mean vector predicted by the controlled neural network. Indicates by parameters The variance scalar of the controlled neural network predictions. The identity matrix indicates that the covariance matrix of this Gaussian distribution is a diagonal matrix, and the variances of each dimension are equal; that is, the features of each dimension are independent, and the boundaries of the three modality pairs are optimized—(text, vision), (vision, acoustic) and (text, acoustic). KL divergence is used to measure the true conditional distribution. and multivariate Gaussian distribution Differences; It is a variational lower bound estimate of mutual information; This is a constant term used to ensure that the lower bound is consistent with the direction of mutual information.

[0109] To preserve cross-modal invariance and ensure the accuracy of the trained model, features based on the relationships between the three modalities and single-modal features are used. With fusion modal features Maximizing mutual information between them generates the mutual information loss function of the fusion layer. The process is as follows:

[0110] S51: Prediction results of fused modal features Z based on formula (11) Perform L2 normalization and obtain the unit vector. ;

[0111] (11);

[0112] In the formula, It is a parameter The neural network, which is derived from Generate pairs The prediction It is the Euclidean norm, which is used to normalize a vector to obtain a vector of unit length.

[0113] S52: Based on formula (12), single-modal features vectors in Perform L2 normalization and obtain the vector. normalized value ;in, ;

[0114] (12);

[0115] S53: Calculate the normalized value using the dot product. and The similarity is obtained by formula (13) to obtain the modal feature similarity score. ;

[0116] (13);

[0117] Based on this similarity score The acquisition of this information allows for full utilization of key modal information and effectively measures the correlation between fused modal features Z and modal representation.

[0118] S54: Obtain the joint loss function according to formula (14) ;

[0119] (14);

[0120] In the formula, for and Similarity score; For all single-modal features Mid-vector With fusion modal features The sum of similarities is used to normalize the score; The desired operation is denoted as for single-modal features. Mid-vector The statistical average;

[0121] S55: Obtain the mutual information loss function of the fusion layer according to formula (15) ;

[0122] (15);

[0123] In the formula, Representing visual modalities The mutual information loss with the fused modal features Z, Representing acoustic modes The mutual information loss with the fused modal features Z, Representing text modality Mutual information loss with fused modal features Z.

[0124] Step Six: Construct a multimodal emotion recognition model;

[0125] S61: Based on the multimodal feature extraction module, graph structure module, multimodal dynamic fusion module, and multimodal mutual information maximization module, a sentiment classifier is added to form the initial recognition model; the sentiment classifier uses the Softmax function for calculation to predict the sentiment label of each sentence; simultaneously, a mutual information loss function including the fusion layer is constructed. Total loss function ;

[0126] To effectively optimize the model parameters, the total loss function is obtained according to formula (16). ;

[0127] (16);

[0128] In the formula, Let cross-entropy be the loss function. For hyperparameters related to mutual information.

[0129] S62: Train the initial recognition model using the training and test sets, and use the total loss function during the training process. The model parameters were optimized to improve the accuracy of emotion recognition, and a multimodal emotion recognition model was obtained after training.

[0130] To improve the accuracy of emotion recognition, the initial recognition model is trained as follows:

[0131] S62-1: Set the maximum number of training epochs, convergence threshold, and other hyperparameters required for training;

[0132] S62-2: Begin end-to-end training of the initial recognition model using the training and test sets; align the raw acoustic features with timestamps from a single dialogue sentence. Original visual features and original text features As input data, it is fed into the initial recognition model to predict and output the sentiment label in the dialogue.

[0133] In each round of training, the following operations are performed: determine whether the learning rate needs to be adjusted in the current round, pass the input data into the initial recognition model, predict and generate sentiment labels, calculate the error between the predicted sentiment labels and the real labels, calculate the loss function and backpropagate to update the parameters of each layer, calculate the difference between the loss function in the current round and the previous round, and compare the difference with the convergence threshold to determine whether convergence has occurred. When convergence occurs, the model parameters in the training process are saved first, and then the next round of training is performed. When convergence fails, the model parameters in the training process are not saved, and the next round of training is performed directly.

[0134] S62-3: After the loss function converges and the best result is obtained or the maximum number of training rounds is reached, the model training is completed. The model parameters after training are saved to obtain the multimodal emotion recognition model.

[0135] Step 7: Perform online emotion recognition;

[0136] Collect a sentence containing the original acoustic features Original visual features and original text features The dialogue data is used as the prediction input data, and the speech features are first extracted by the multimodal feature extraction module. Visual features and text features Then the graph structure module calculates the edge weights. Simultaneously, by propagating features through intramodal edges, long-distance intramodal dependencies are captured; feature interaction is achieved through intermodal edges; and the final feature representation of multimodal nodes is output. Next, the multimodal dynamic fusion module aggregates intramodal and intermodal contextual information through convolutional operations, and finally fuses the multimodal contextual features into the dynamic semantic space, outputting the fused modal features of each sentence. Then, the fused modal features Input the sentiment classifier, which uses the Softmax function to calculate and predict and output the sentiment label for each utterance.

[0137] Validation experiments were conducted on the IEMOCAP and MELD datasets, effectively demonstrating the effectiveness of the multimodal situation recognition method in this invention. Furthermore, the confusion matrix obtained on the IEMOCAP dataset is as follows: Figure 2 As shown, the diagonal values ​​of the matrix confirm that the multimodal emotion recognition model in this invention can accurately identify the true emotions of most samples, while the minimal off-diagonal error indicates a low overall level of confusion, demonstrating that the multimodal emotion recognition model in this invention can effectively distinguish different emotion categories. Overall, the method proposed in this invention improves emotion representation through cross-modal complementary fusion, and its performance in emotion recognition tasks significantly outperforms existing multimodal fusion methods.

[0138] To address the shortcomings of existing multimodal emotion recognition technologies, such as insufficient deep cross-modal dependency mining, poor dynamic interaction modeling, and incomplete long-range context capture, this invention captures the constantly evolving interaction and dependency relationships in the context through dynamic graph networks, and optimizes multimodal fusion by employing cross-modal mutual information maximization and single-modal mutual information maximization strategies. Specifically, this invention proposes a multimodal dialogue emotion recognition method based on graph dynamic fusion and mutual information maximization. First, dialogue data containing acoustic, visual, and textual information is collected, and the original features of each modality are extracted separately. Then, a sample dataset is constructed based on timestamp alignment, providing a reliable data foundation for the subsequent training of the recognition model. Second, a multimodal feature extraction module is constructed using OpenSmile, Roberta, and DenseNet, which can be easily used to encode speech, vision, and text separately, thus efficiently obtaining speech, visual, and textual features. Next, each utterance is represented as a cross-modal node, and a dialogue-based graph structure module is constructed. This module can comprehensively capture multimodal information in the dialogue, avoiding the limitations of a single modality, and effectively promoting cross-modal feature interaction and enhancing feature complementarity. Subsequently, a multimodal dynamic fusion module is constructed based on an improved graph convolutional layer and a dynamic gating mechanism. This module can dynamically aggregate contextual information inside and outside the modality, and achieve deep fusion of multimodal features in the dynamic semantic space through hierarchical stacking, outputting the fused modal features of each utterance. Furthermore, based on Barber... The Agakov lower bound estimation method constructs a multimodal mutual information maximization module, which can implement mutual information maximization constraints between modalities and between single modalities and fused features, and generates a mutual information loss function for the fusion layer, enabling optimization of model parameters through end-to-end training. Finally, a sentiment classifier is introduced, which can efficiently and accurately predict the sentiment label of each dialogue through the calculation process of the Softmax function.

[0139] This method is simple to implement, low in cost, and highly accurate. It effectively reduces redundant information and enhances cross-modal complementarity by using graph structure modeling for cross-modal dynamic interaction fusion and mutual information constraints to enhance feature complementarity, thus significantly improving the accuracy and robustness of dialogue emotion recognition.

Claims

1. A multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization, characterized in that, The method comprises the following steps: Step one: obtaining a multi-modal dialogue data set; The process of obtaining the multi-modal dialogue data set is as follows: From a dialogue dataset containing acoustic, visual and text information, original acoustic features , original visual features and original text features are extracted respectively, and the timestamps of the original acoustic features , original visual features and original text features are aligned to form a sample dataset; the sample dataset is divided into a training set and a test set according to a set proportion; Step two: constructing a multi-modal feature extraction module for extracting multi-modal features; Step three: constructing a graph structure module; constructing an undirected graph according to the number of utterances in the dialogue; initializing the node embedding in the graph, connecting all nodes in the same mode to form intra-modal edges to capture intra-modal long-range dependencies, and connecting different modal nodes in the same utterance to form inter-modal edges to facilitate cross-modal information interaction, thereby forming a dialogue-based graph structure module; The graph structure module is configured to calculate weights of edges Meanwhile, the features are propagated through the intra-modal edges to capture the long-range dependencies within the modal, and the features are interacted through the inter-modal edges to output the final feature representation of the multi-modal nodes ; Step four: constructing a multi-modal dynamic fusion module for fusing multi-modal features; Step five: constructing a multi-modal mutual information maximization module; constructing a mutual information maximization module including a mean prediction network and a variance prediction network based on a Barber-Agakov lower bound estimation method, the mutual information maximization module is used for mutual information lower bound estimation, and a fusion layer mutual information loss function is generated based on mutual information maximization between features of three modalities and single modal features and fusion modal features ;​ Step six: constructing a multi-modal emotion recognition model; S61: on the basis of the multi-modal feature extraction module, the graph structure module, the multi-modal dynamic fusion module and the multi-modal mutual information maximization module, a sentiment classifier is added to constitute an initial recognition model; at the same time, a total loss function containing a fusion layer mutual information loss function is constructed ; S62: training the initial recognition model using the training set and the test set, wherein a total loss function is used in the training process Optimizing the model parameters, and obtaining the multi-modal sentiment recognition model after training; Step seven: online recognition of emotion; collect a dialogue data containing original acoustic features , original visual features and original text features as prediction input data; input the prediction input data into the multi-modal emotion recognition model for recognition, and output emotion labels.

2. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 1, characterized in that, In step two, the process of constructing the multi-modal feature extraction module is as follows: The multi-modal feature extraction module is constructed based on OpenSmile, Roberta and DenseNet; OpenSmile, Roberta and DenseNet are used to encode the raw acoustic features , raw visual features and raw textual features respectively, and obtain the speech features , visual features and textual features according to formula (1), formula (2) and formula (3) respectively. （1）； （2）； （3）； In the formula, are parameters of the OpenSmile tool, are parameters of the DenseNet model, are Roberta model parameters.

3. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 2, characterized in that, In step three, when constructing the undirected graph, each utterance is represented by three nodes corresponding to the acoustic, visual and textual modalities, and in addition, the weights of the edges are calculated according to the cosine similarity function according to formula (4) ; （4）； where, is the embedding vector of node ; is the embedding vector of node ; is the cosine similarity; is used to convert the similarity to an angle, normalized to the [0, 1] interval.

4. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 3, characterized in that, In step four, the process of constructing the multi-modal dynamic fusion module is as follows: Based on the improved graph convolution layer and the introduction of a dynamic gating mechanism, a multi-modal dynamic fusion module is constructed, the multi-modal dynamic fusion module is a multi-layer structure, and the final feature representation of the multi-modal node is obtained As input, the feature aggregation within the mode is first realized by residual connection and dynamic weight in each layer to obtain the graph convolution output As shown in formula (5), on the basis of feature aggregation, the enhanced output is obtained by dynamically fusing the cross-modal context information through the gating mechanism As shown in formula (6), finally, through the stacking of the multi-layer structure, the multi-modal context features are fused into the dynamic semantic space, and the fused modal features of each sentence are output ; （5）； In the formula, is the graph convolution output of the i-th layer in the multi-modal dynamic fusion module; is the graph convolution output of the i-th layer in the multi-modal dynamic fusion module; is a nonlinear activation function; is a residual coefficient; is the residual state after graph convolution of the i-th layer; is the residual state after graph convolution of the i-th layer; is a renormalized graph convolution matrix, , is a degree matrix, is an adjacency matrix; is an initial multi-modal feature, obtained by , , initialization; is an inter-layer gating coefficient, , is a hyperparameter; is a learnable weight matrix; is a unit mapping matrix; （6）； In the formula, is the enhanced output of the i-th layer in the multi-modal dynamic fusion module; is the enhanced output of the i-th layer in the multi-modal dynamic fusion module; is the gating signal.

5. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 4, characterized in that, In step four, the process of dynamically fusing cross-modal context information through a gating mechanism is as follows: S41: Calculate the gate signal according to formula (7) , obtained the Layer nodes Aggregation characteristics; （7）； wherein is a sigmoid function; is a gating weight matrix; is a gating output of the layer; is an augmented hidden state of the layer; is a gating bias term; S42: generate the candidate memory cell according to formula (8) to store the first context information of the layer; （8）； wherein is a memory cell weight matrix; is a memory cell bias term; S43: update the gate by and the forget gate to update the up and down information, and obtain the updated memory cell according to formula (9) ; （9）； S44: by the output gate performing output of selective information; denotes element-wise multiplication.

6. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 5, characterized in that, In step five, the mutual information maximization module is used to perform the mutual information lower bound estimation, and the mutual information is obtained according to formula (10) ; （10）； where, is a multivariate Gaussian distribution, , denotes the mean vector predicted by the neural network controlled by parameters , denotes the variance scalar predicted by the neural network controlled by parameters , is an identity matrix, indicating that the covariance matrix of this Gaussian distribution is a diagonal matrix with equal variance in each dimension; is the KL divergence, used to measure the difference between the true conditional distribution and the multivariate Gaussian distribution ; is the variational lower bound estimate of mutual information; is a constant term to ensure that the lower bound is consistent with the direction of mutual information.

7. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 6, characterized in that, In step five, based on the features between the three modes and the single-mode features With fusion modal features Maximizing mutual information between them generates the mutual information loss function of the fusion layer. The process is as follows: S51: According to formula (11), the prediction result of the fusion modal feature Z L2 normalization is performed, and a unit vector is obtained ; （11）； In the formula, is a neural network with parameters is the Euclidean norm, and a unit length vector is obtained by normalizing the vector.​ S52: L2-normalize the vector in the single-modal feature according to equation (12) and obtain the normalized value of the vector ;​ （12）； S53: Calculate the normalized value by dot product and The similarity of the modal features is obtained according to formula (13) ; （13）； S54: Obtain the joint loss function according to formula (14) ; （14）； wherein is and a similarity score; is a sum of similarities of vectors in all single modality features to fused modality features for normalizing the score; is an expected operation, representing a statistical average of vectors in single modality features ; S55: Obtain the fusion layer mutual information loss function according to formula (15) ; （15）； wherein represents the visual modality mutual information loss with the fused modality features Z, represents the acoustic modality mutual information loss with the fused modality features Z, represents the textual modality mutual information loss with the fused modality features Z.

8. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 7, characterized in that, In step six S61, the total loss function is obtained according to formula (16) ; （16）； wherein is the cross-entropy loss function, is a hyperparameter related to mutual information.

9. The multi-modal sentiment recognition method based on graph dynamic fusion and mutual information maximization according to claim 8, characterized in that, In S62 of step six, the process of training the initial recognition model is as follows: S62-1: setting the maximum number of training rounds, the convergence threshold and other hyperparameters required for training; S62-2: Begin end-to-end training of the initial recognition model using the training and test sets; align the raw acoustic features with timestamps from a single dialogue sentence. Original visual features and original text features As input data, it is fed into the initial recognition model to predict and output the sentiment label in the dialogue. In each round of training, the following operations are performed: determining whether the learning rate needs to be adjusted in the current round, inputting the data into the initial recognition model, predicting the generated emotion label, calculating the error between the predicted generated emotion label and the real label, calculating the loss function and updating the parameters of each layer through back propagation, calculating the difference between the loss functions of the current round and the previous round, comparing the difference with the convergence threshold to determine whether convergence is achieved, when convergence is achieved, the model parameters in the training process are saved first, and then the next round of training is performed, when convergence is not achieved, the model parameters in the training process are not saved, and the next round of training is directly performed; S62-3: after the loss function converges and the best result is obtained or the maximum number of training rounds is reached, the model training is completed, the model parameters after the training are saved, and a multi-modal emotion recognition model is obtained.

Images