Multimodal emotion recognition method and system based on domain generalization and graph neural network

Abstract

The invention discloses a multi-modal emotion recognition method and system based on field generalization and a graph neural network, and relates to the technical field of emotion recognition. According to the technical key points, the method comprises the following steps: acquiring a data set, wherein the data set comprises three types of modal data: a text modal, a voice modal and a visual modal; performing feature extraction on the three types of modal data to obtain text features, voice features and image features; performing field generalization by taking each mode as a field, namely dividing features extracted by each mode into two parts, one part is used for extracting intra-domain invariant features, and the other part is used for extracting inter-domain invariant features; fusing the intra-domain invariant features and the inter-domain invariant features by using a graph neural network to obtain global information and local context information at the same time; inputting the fused features into a classification model for training; inputting the to-be-detected data into the trained classification model for classification, and obtaining an emotion recognition result. According to the method, the emotion recognition performance and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and particularly relates to a multi-modal emotion recognition method and system based on domain generalization and graph neural network. Background Art

[0002] Emotion is an inherent emotion of human beings, which guides their behaviors and indicates potential thought processes. Therefore, understanding and recognizing emotions is crucial for developing artificial intelligence technologies (e.g., personal digital assistants) that can interact directly with humans. In conversations among many people, the emotions experienced and expressed by each person constantly fluctuate. With the rapid development of information technology and the popularity of social media, people generate a large amount of multi-modal data in their daily lives, including text, images, audio, etc. These data contain rich emotional information, such as the emotions expressed in users' posts, comments, pictures, and videos on social media. Therefore, the automatic recognition and understanding of emotions have become an important research direction in the fields of natural language processing and artificial intelligence. However, due to the complexity and heterogeneity of multi-modal data, single-modal emotion recognition methods can no longer meet the needs of dialogue emotion analysis, so multi-modal emotion recognition methods are required to better understand and analyze the emotional content in conversations.

[0003] In addition, the analysis of multi-modal dialogue data also has great application value in social life. For example, in public opinion analysis, malicious public opinion manipulation on the Internet may pose a threat to user safety. By comprehensively analyzing the speeches of Internet users and taking corresponding measures, the Internet public opinion environment can be effectively improved. In intelligent customer service, the system can dynamically adjust the response strategy according to the emotions expressed in the user's speech content and tone, improving the user experience. In the medical industry, an emotion recognition system can assist in diagnosing and treating patients, especially in the communication with patients with abnormal expressions and body movements, such as autistic patients, providing important diagnostic and treatment support.

[0004] Currently, for emotion recognition in conversations, mainstream methods usually use global information and local context information to predict the emotion labels of each sentence. Global information can be captured by modeling cross-modal interactions at the dialogue level, while local information is usually inferred from the time information or emotion changes of the speaker. However, the way of extracting single-modal features in existing methods is not suitable for subsequent multi-modal fusion, which leads to limitations in emotion recognition performance.

[0005] Multiple interlocutors, dynamic interactions, and context dependence pose challenges to the task of conversational emotion recognition (ERC). First, in multi-participant conversations, the emotional states of each person can influence and change each other. Relying solely on the independent emotion recognition of each participant is insufficient, and it is necessary to model the interaction relationships among the participants. For example, a person's mood may be affected by the responses of others, even triggering a chain reaction. How to capture this emotional interaction process among participants is the key to ERC modeling.

[0006] Second, conversation is a dynamic interaction process, and the emotional states of the participants also change continuously as the conversation progresses. Relying solely on the information of the current utterance is far from enough, and it is necessary to model the context information of the entire conversation history. The emotional attitudes, interaction patterns, etc. implicit in past utterances may all affect the current emotional state. How to effectively model this temporal dependence relationship is also a major challenge.

[0007] Furthermore, conversations contain rich context information, including the identities, relationships, topics, environments, etc. of the participants. These context factors are all closely related to emotions, but it is not an easy task to model them comprehensively. In contrast, most existing studies mainly focus on the text modality and ignore other potentially valuable information sources, such as speech, vision, etc. How to integrate multi-modal information to enhance the modeling of conversational emotions is also a direction worthy of exploration in the future. In the ERC task, efforts are being made to model the conversational context and mainly focus on the text modality.

[0008] Multi-modal machine learning methods have become popular because they can address the limitations of single-modal methods in capturing complex real-world phenomena. It is recognized that human perception and understanding are influenced by the integration of multiple sensory inputs. There are several notable methods aiming to apply the advantages of multi-modal to various applications. CMN combines features from different modalities by directly connecting them and uses gated recurrent units (GRUs) to model context information. ICON extracts multi-modal conversation features and uses global memory hierarchical modeling of emotion influence, thus improving the performance of utterance-video emotion recognition. ConGCN models utterances and speakers as nodes in a graph and captures context dependence and speaker dependence as edges. However, ConGCN only focuses on text and audio features and does not consider other modalities. In addition, MMGCN is a model based on graph convolutional network (GCN) that can effectively capture long-range context information and multi-modal interaction information. However, existing models do not cover the temporal aspect in conversations.

[0009] In the past few years, there has been an increasing interest in representing data as graphs. However, the complexity of graph data poses challenges to traditional neural network models.

[0010] Traditional vector-based neural network models face many challenges when dealing with graph data. First, graph data is highly irregular and complex, making it difficult to represent with fixed-size vectors. The number of nodes and edges in a graph can vary greatly, making it hard to directly input into a neural network. Second, the graph structure contains rich topological information and neighborhood dependencies, which are difficult to capture through simple vector encoding. Finally, graph data is often highly dynamic and heterogeneous, making it hard to describe with a unified model framework.

[0011] To better utilize the structural characteristics of graph data, researchers have proposed new deep learning models such as graph neural networks (GNNs). Through convolutional and pooling operations defined on the graph structure, GNNs can effectively extract the features of nodes and edges in the graph and model the complex topological relationships between them. At the same time, GNNs also have good scalability and flexibility, and can be applied to various types of graph data.

[0012] Compared with traditional neural networks, GNNs have shown significant advantages in graph data modeling. First, GNNs can better capture the structural information of nodes and their neighborhoods in the graph, resulting in richer feature representations. Second, GNNs can model the complex interaction relationships between nodes by leveraging the topological information of the graph, which is very valuable in many practical applications. Third, GNNs have good interpretability, can clearly show the mechanisms of feature extraction and information propagation, and help in-depth understanding of model behavior.

[0013] When faced with complex interdependencies between modalities, GNNs are a more effective way to utilize the potential of multimodal datasets. The advantage of GNNs lies in their ability to capture and model intra-modal and multimodal interactions. This flexibility makes them an attractive choice for multimodal learning tasks.

[0014] In addition, graph convolutional networks have also developed rapidly. Since they can make good use of the topological structure characteristics between node features to optimize node features, improve the classification accuracy, and have fewer model parameters for easy training.

[0015] However, existing models mainly focus on modality-level interactions and do not consider the temporal aspects within the graph.

[0016] Domain generalization is one of the important methods to solve cross-domain sentiment analysis. As a core task in natural language processing, sentiment analysis plays an important role in many practical applications, such as public opinion monitoring, customer service, product review analysis, etc. However, due to differences in language expression habits in different domains or scenarios, it is difficult for sentiment analysis models based on a single domain to maintain good generalization performance in cross-domain scenarios.

[0017] Traditional solutions usually fine-tune and adapt the model with a large amount of labeled data. However, this method has two main problems: 1) For newly emerging fields, it is often necessary to re-collect and label a large amount of data, resulting in high costs; 2) The model tends to overfit the characteristics of a specific field and is difficult to maintain good generalization ability.

[0018] Domain generalization techniques, on the other hand, provide a more efficient and robust solution to solve this problem. Its core idea is to train a general sentiment analysis model on a limited training dataset that can generalize to multiple domains. This requires the model to learn sentiment expression features shared across domains without overfitting the special patterns of a specific domain. This method improves the generalization ability of the model by leveraging the data relationship between the source domain and the target domain. Existing domain generalization methods mainly include data augmentation, domain-invariant feature learning, and meta-learning techniques. Most existing work focuses on some specific applications, such as computer vision and reinforcement learning. Among them, the cross-domain invariant feature learning method mainly focuses on the multi-modal sentiment recognition task.

[0019] Domain-invariant feature learning is a popular DG strategy that aims to learn representations that remain invariant across different domains, thus facilitating cross-domain generalization. The popularity and effectiveness of domain-invariant learning naturally prompt the search for the underlying principles behind this method: What are the features of domain-invariant learning, and how to further improve its performance in DG.

[0020] Traditional sentiment analysis models usually perform poorly in new domains because they cannot well adapt to the data characteristics of new domains. Therefore, how to design a model that can generalize across different domains is an important research topic. Summary of the Invention

[0021] In view of this, the present invention proposes a multi-modal sentiment recognition method and system based on domain generalization and graph neural network to improve the performance and robustness of sentiment recognition.

[0022] According to one aspect of the present invention, a multi-modal sentiment recognition method based on domain generalization and graph neural network is proposed, and the method includes:

[0023] Obtain a dataset, where the dataset includes three types of modal data: text modality, speech modality, and visual modality;

[0024] Extract features from the three types of modal data to obtain text features, speech features, and image features;

[0025] Perform domain generalization with each modality as a domain, that is, divide the features extracted from each modality into two parts, one part is used to extract intra-domain invariant features, and the other part is used to extract inter-domain invariant features;

[0026] Use a graph neural network to fuse the in - domain invariant features and the inter - domain invariant features to simultaneously obtain global information and local context information;

[0027] Input the fused features into a classification model for training;

[0028] Input the data to be tested into the trained classification model for classification to obtain the sentiment recognition result.

[0029] Furthermore, for the text modality, use the RoBERTa model for pre - training and fine - tuning to extract text features; for the speech modality, use the OpenSmile tool to extract speech features; for the visual modality, use the DenseNet model to extract features of facial images.

[0030] Furthermore, the domain generalization with each modality as a domain includes:

[0031] Adopt a distillation algorithm to learn in - domain invariant features, including: the teacher network uses the Fourier phase information and class labels as input and output respectively to obtain Fourier phase information features for classification;

[0032] Adopt the CORAL alignment algorithm to learn inter - domain invariant features;

[0033] Then determine the objective function of domain generalization as follows:

[0034]

[0035] In the formula, The loss function corresponding to learning in - domain invariant features; The loss function corresponding to learning inter - domain invariant features; Represents the loss function corresponding to the introduced regularization term; θ f ,θ c Are parameters to be solved, representing the learnable parameters of the feature extraction network G f And the classifier G c respectively; Represents the expectation of the distribution p tr of the training feature data; Represents the cross - entropy loss; Represents the mean square error loss; Represents the true value of the teacher network; (x, y) represents the training feature data, x represents the feature data, and y is the corresponding label; Represents the student network feature extractor; z 2 =G f (x); λ 1 、λ 2, λ 3 is a hyperparameter; min represents minimization;

[0036] Use an optimization algorithm to iteratively solve the above objective function and update the model parameters θ f , θ c to optimize the invariant features z within the domain 1 and the invariant features z between domains 2 .

[0037] Furthermore, the loss function for learning the invariant features between domains is:

[0038]

[0039] In the formula, C i , C j is the covariance matrix composed of the sample features between different domains; ||·|| F is the norm; N represents the number of feature samples; i, j represent the sample indices between different domains;

[0040] The loss function corresponding to the introduced regularization term is:

[0041]

[0042] In the formula, d(·,·) is the distance function.

[0043] Furthermore, the fusion of the invariant features within the domain and the invariant features between domains using the graph neural network includes:

[0044] Connect the extracted invariant features within the domain and the invariant features between domains to obtain the connected features

[0045] Input the connected features z into the heterogeneous graph neural network to extract node features;

[0046] For different node features, use the attention mechanism for fusion to obtain the fused features.

[0047] Furthermore, the formula for inputting the connected features z into the heterogeneous graph neural network to extract node features is:

[0048]

[0049] In the formula, Θ root , Θ r are the learned parameters; |N r (i)| is the normalization constant; z i and z jThe i-th and j-th features of the concatenated feature z; R represents the set of relationships.

[0050] Furthermore, the formula for fusing different node features using the attention mechanism is:

[0051] H′ i = W 1 x′ i + ∑α i,j W 2 x′ j

[0052] In the formula, is the attention coefficient; W 1 、W 2 、W 3 and W 4 are all learnable weight matrix parameters; x' i 、x' j represent two node features obtained from the heterogeneous graph neural network; m represents the dimension of the feature space, and softmax represents the activation function.

[0053] Furthermore, inputting the fused features into the classification model for training includes: the classification model uses a fully connected neural network; inputting the fused features into a fully connected layer; using the softmax activation function to convert the output of the fully connected layer into a probability distribution, and taking the maximum value in the probability distribution as the predicted sentiment label.

[0054] Furthermore, the sentiment recognition results include: happy, sad, neutral, angry, excited, frustrated, disgusted, fearful, surprised.

[0055] According to another aspect of the present invention, a multi-modal sentiment recognition system based on domain generalization and graph neural network is proposed. The system includes:

[0056] A data acquisition module configured to acquire a data set, where the data set includes three types of modal data: text modality, speech modality, and visual modality;

[0057] A feature extraction module configured to extract features from the three types of modal data to obtain text features, speech features, and image features;

[0058] A domain generalization module configured to perform domain generalization on each modality as a domain, that is, divide the features extracted from each modality into two parts, one part is used to extract intra-domain invariant features, and the other part is used to extract inter-domain invariant features;

[0059] A feature fusion module configured to use a graph neural network to fuse the intra-domain invariant features and the inter-domain invariant features to simultaneously obtain global information and local context information;

[0060] A model training module configured to input the fused features into a classification model for training;

[0061] An emotion recognition module configured to input the data to be measured into the trained classification model for classification to obtain an emotion recognition result.

[0062] The beneficial technical effects of the present invention are:

[0063] The present invention provides a multi-modal emotion recognition method and system based on domain generalization and graph neural network, including a new model - TL-RGCN. The key modules included are a domain generalization module and a graph neural network module. The former performs cross-modal interaction by taking the features extracted from a single modality as a domain respectively, so that the extracted features are more adaptable to subsequent multi-modal feature fusion; the latter learns global and local information through the graph neural network to capture the emotions of the interlocutors. Experimental results on three datasets, namely IEMOCAP, CMU-MOSEI, and MELD, show that the method of the present invention has greatly improved the performance of multi-modal emotion recognition. Description of the Drawings

[0064] By referring to the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary rather than restrictive manner, wherein:

[0065] Figure 1 is a flowchart of a multi-modal emotion recognition method based on domain generalization and graph neural network according to an embodiment of the present invention.

[0066] Figure 2 is a flowchart block diagram of a multi-modal emotion recognition method based on domain generalization and graph neural network according to an embodiment of the present invention.

[0067] Figure 3 is a block diagram of a distillation algorithm in an embodiment of the present invention.

[0068] Figure 4 is an exemplary diagram of a classification confusion matrix on the IEMOCAP dataset in an embodiment of the present invention.

[0069] Figure 5 is a schematic structural diagram of a multi-modal emotion recognition system based on domain generalization and graph neural network according to an embodiment of the present invention. Detailed Embodiments

[0070] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and then implement the present invention, and not to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to fully convey the scope of the present disclosure to those skilled in the art.

[0071] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, a device, an equipment, a method or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: completely hardware, completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. In the present invention, it should be understood that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0072] The present invention proposes a multi-modal sentiment recognition method and system based on domain generalization and graph neural network, that is, a multi-modal sentiment recognition framework named TL-RGCN is proposed. This framework mainly includes three modules: First is the feature extraction module. For the text modality, the RoBerta model is used in the present invention to extract features; for the speech modality, the OpenSmile toolkit is used in the present invention to extract features; for the visual modality, the DenSeNet convolutional network is used in the present invention to extract facial features. Secondly is the domain generalization module. The present invention generalizes different domains by taking each modality as a domain, mainly dividing the features of each modality into two parts, one part is used to extract intra-domain invariant features, and the other part is used to extract inter-domain invariant features. Finally is the graph neural network module, which mainly extracts global information and local information through RGCN, so as to improve the performance of the sentiment recognition of the interlocutor's utterance. Further experiments are carried out on three datasets, namely IEMOCAP, CMU-MOSEI and MELD. The experiments show that the performance and robustness of the present invention have been significantly improved compared with the existing methods; ablation experiments are carried out on the domain generalization module and the graph neural network module, and the experiments prove the effectiveness of the two modules.

[0073] An embodiment of the present invention proposes a multi-modal sentiment recognition method based on domain generalization and graph neural network, as Figure 1 shown, this method includes:

[0074] S1. Obtain a dataset, where the dataset includes three types of modal data: text modality, speech modality, and visual modality;

[0075] S2. Extract features from the three types of modal data to obtain text features, speech features, and image features;

[0076] S3. Generalize each modality as a domain, that is, divide the features extracted from each modality into two parts, one part is used to extract intra-domain invariant features, and the other part is used to extract inter-domain invariant features;

[0077] S4. Use a graph neural network to fuse the intra-domain invariant features and the inter-domain invariant features to simultaneously obtain global information and local context information;

[0078] S5. Input the fused features into a classification model for training;

[0079] S6. Input the data to be tested into the trained classification model for classification to obtain the sentiment recognition result.

[0080] Figure 2 shows the model framework diagram of the present invention. Given a multi-speaker conversation C[u 1 , u 2 , …, u N consisting of N utterances, let S denote the respective speaker sets. Each utterance u i is associated with three modalities, including text (t), audio (a), and vision (v), which can be respectively represented as Using local and global context representations, the goal of the sentiment recognition task in the conversation is to predict the label [z 1 , z 2 , … z M of each utterance from a set of M predefined emotion labels Z.

[0081] The present invention will be described in detail below.

[0082] First, in S1, a data set is obtained, and the data set includes three modality data: text modality, speech modality, and visual modality.

[0083] According to the embodiments of the present invention, the three data sets used store the data of the three modalities (text modality, speech modality, and visual modality) in different files respectively and with labels. Specifically, for audio data, the moviepy tool in Python can be used for audio extraction; for text data, the SpeechRecognition library in Python can be used to convert the extracted audio data into text; for visual data, the OpenCV library in Python can be used to extract the data of each frame or key frame; finally, only need to store the above three kinds of data in different folders and make them corresponding through the naming rule.

[0084] Then, in S2, feature extraction is performed on the three modality data to obtain text features, speech features, and image features.

[0085] According to an embodiment of the present invention, for text-modal data, the present invention uses the RoBERTa model for pre-training and fine-tuning to extract text features; for speech-modal data, the OpenSmile tool is used for feature extraction; for video-modal data, the DenseNet model is used for feature extraction of facial expressions.

[0086] RoBERTa is improved and optimized based on the BERT (Bidirectional Encoder Representations from Transformers) model. It is pre-trained on large-scale data, using a larger dataset and a longer training time, thereby improving the model performance and generalization ability. The structure of the RoBERTa model is basically the same as that of BERT, consisting of multiple Transformer blocks. Each block contains a multi-head self-attention mechanism, a fully-connected layer, and a residual connection, where the multi-head self-attention mechanism can achieve context-related encoding, and the fully-connected layer maps the encoded vector into a new vector space. Specifically, the structure of the RoBERTa model can be expressed as formula (1).

[0087] h l = TransformerBlock(h l-1 ) (1)

[0088] The input vector of the RoBERTa model is composed of three parts: token embedding, position embedding, and segment embedding. Among them, the token embedding converts the input text into a corresponding vector representation, the position embedding represents the position information of each word in the input text, and the segment embedding represents the boundary information of each sentence in the input text. Compared with the BERT model, the RoBERTa model uses more and larger text data for training, including web pages, forums, books, news, etc. on the Internet. This enables the model to better capture the context relevance in natural language, thereby improving the model performance.

[0089] OpenSmile is an open-source audio feature extraction toolkit for extracting various audio features from audio signals. The speech features extracted in this invention include spectral features, voice quality features, and prosodic features. Mel Frequency Cepstral Coefficients (MFCC) is one of the most widely used spectral features, and its proposal is closely related to the auditory characteristics of the human ear. To extract MFCC features, first, it is necessary to perform data preprocessing on the original speech signal. The preprocessing process usually includes a series of operations such as filtering, denoising, pre-emphasis, framing, and windowing. Secondly, perform a discrete Fourier transform on the processed data to obtain its spectrum. Then, use the Mel filter bank to process the signal spectrum and perform a logarithmic operation on the output result of the Mel filter. Finally, perform a discrete cosine transform on the logarithmic energy output by the filter to obtain the MFCC features. Voice quality features are usually used to measure the clarity of sound and are a subjectively defined evaluation index by humans. However, since humans often experience phenomena such as choking, gasping, and trembling when affected by emotional fluctuations, voice quality features are also often used in the field of speech emotion recognition. Common voice quality features include glottal parameters, jitter and shimmer, formant frequency and its bandwidth, etc. Prosodic features can reflect the degree of changes such as the stress, tempo, and length of speech, so prosodic features are also often used in the field of speech emotion analysis. Some of the prosodic features are related to the fundamental frequency, such as the fundamental frequency and its mean and variance; some are related to the energy, such as the short-time average energy, amplitude, and energy change rate; and some are related to the duration, such as the short-time zero-crossing rate, speech rate, and duration.

[0090] DenseNet (Densely Connected Convolutional Networks) is a deep convolutional neural network architecture proposed by Gao Huang et al. in 2017. Different from traditional Convolutional Neural Networks (CNN), DenseNet adds direct connection paths in the network through dense connections, thus achieving more compact feature transfer and reuse. As a densely connected neural network structure, through its unique design, it effectively improves the efficiency of feature transfer and the utilization efficiency of parameters, providing an efficient network architecture for tasks in image processing and other fields. Use DenseNet to obtain the feature vector representation of the visual modality.

[0091] Then, in S3, each modality is used as a domain for domain generalization, that is, the features extracted from each modality are divided into two parts, one part is used to extract within-domain invariant features, and the other part is used to extract between-domain invariant features.

[0092] According to the embodiments of the present invention, in order to maintain fairness, the extracted features are split into two parts. One part is used for learning internally-invariant features within the domain, and the other part is used for learning inter-domain features. At the same time, in order to ensure the diversity of features, a regularization term is proposed to make the difference between the two types of features as large as possible.

[0093] 1) Learning internally-invariant features: Internally-invariant features are generated within the domain and are not affected by other domains, mainly capturing the intrinsic semantic information of the data. To obtain internally-invariant features, a simple distillation framework is mainly used for learning, as Figure 3 shown.

[0094] Specifically, the teacher network uses the Fourier phase information and class labels as input and output respectively to obtain the Fourier phase information features for classification. The Fourier transform F(x) of the single-channel two-dimensional data x is expressed as formula (2).

[0095]

[0096] where u and v are exponents. H and W are the height and width respectively. The Fourier transform can be effectively calculated using the FFT algorithm. The phase information is expressed as formula (3).

[0097]

[0098] where R(x) and I(x) represent the real part and imaginary part of F(x) respectively. For data with multiple channels, the Fourier transform of each channel is calculated separately to obtain the corresponding phase information.

[0099] Although this distillation method introduces additional training costs during training, it can reduce unnecessary FFT calculations during prediction, ensuring that the entire prediction process can be directly carried out end-to-end. The objective function of this method is formula (4).

[0100]

[0101] where is the cross-entropy loss, is the mean squared error loss, representing the difference between the predicted value of the student network and the true value of the teacher network, λ 1 is a hyperparameter, and are the learnable parameters of the feature extractor and classifier of the student network respectively. represents the expectation, p tr represents the distribution of the training data, The true values (x, y) representing the teacher network represent the training data, where x represents the data and y is the label corresponding to x.

[0102] During the training process, as the parameters and are continuously updated, the cross-entropy loss and the mean squared error loss are continuously reduced, and finally the output z of the feature extraction network 1 will be continuously optimized to obtain Equation (5).

[0103]

[0104] 2) Learning domain-invariant features: It is completed by using the CORAL alignment algorithm, and its loss function formula is shown in Equation (6):

[0105]

[0106] where C i , C j are the covariance matrices composed of the sample features between different domains, ||·|| F is the norm; N represents the number of samples; i, j represent the sample indices between different domains; among them, the covariance matrix C j is obtained in the same way.

[0107] By optimizing the alignment loss the model will learn to make the feature representations C i and C j as close as possible, that is, by minimizing the loss, the feature extraction network can output domain-invariant features to obtain Equation (7).

[0108] z 2 = G f (x) (7)

[0109] where G f (·) is the feature extractor and x is the input feature.

[0110] 3) To better obtain diverse invariant features and reduce the repetition and redundancy between features, it is expected that these two types of invariant features are as different as possible. The present invention achieves this purpose through an additional regularization term.

[0111]

[0112] where d(·,·) is the distance function, and here the distance function can use the L2 distance function:

[0113] In summary, the determined overall objective function is shown in Equation (9):

[0114]

[0115] Among them, λ 1 , λ 2 , λ 3 are hyperparameters, and the parameters θ f , θ c are the learnable parameters of the feature extraction network G f and the classifier G c respectively.

[0116] Use the stochastic gradient descent algorithm - SGD to calculate the gradients of various losses, thereby updating the model parameters θ f , θ c , and finally achieve the purpose of optimizing the in - domain invariant feature z 1 and the inter - domain invariant feature z 2 .

[0117] Then, in S4, use the graph neural network to fuse the in - domain invariant feature and the inter - domain invariant feature to simultaneously obtain global information and local context information.

[0118] According to the embodiments of the present invention, in cross - domain sentiment analysis, the feature extraction of a single modality is a crucial step. However, the features of these single modalities may have different distributions and expressions, resulting in difficulties in the multi - modality fusion process. Therefore, in order to better adapt the extracted single - modality features to the subsequent multi - modality fusion module, a graph neural network module is proposed. This module uses a graph structure to model the relationships between different modalities, thereby realizing the adaptation and fusion of single - modality features. Specifically, represent the features of a single modality as nodes in a graph, and use the graph neural network to learn the representations between nodes to capture the correlations and semantic information between modalities. In this way, the features of a single modality can be better utilized and seamlessly integrated into the subsequent multi - modality sentiment analysis task, thereby improving the performance and generalization ability of the model.

[0119] The role of the graph neural network module is to extract global information and local context information and fuse the features extracted by the domain generalization module. Global information can be captured by modeling cross - modal interactions at the dialogue level, while local information is usually inferred from the time information or emotional changes of the speaker.

[0120] Specifically, first, connect the in - domain invariant feature z 1 and the inter - domain invariant feature z 2 to obtain z, as shown in formula (10).

[0121]

[0122] After that, each sentence is taken as a node, and the order of the spoken words is taken as the edge to construct a directed graph. It includes the relationship R within the same speaker intra and the relationship R between different speakers inter , then the relationships in formulas (11) and (12) hold.

[0123]

[0124] Among them, S 1 and S 2 respectively represent interlocutor 1 and interlocutor 2, and z i represents the node feature, that is, the i-th feature of formula (10).

[0125] Then, through an RGCN (Relational Graph Convolutional Network) module, RGCN is a specific transformation that helps accumulate the relationships of adjacent nodes by normalization and according to the types and directions of the edges appearing in the graph. RGCN (Relational Graph Convolutional Network) is a graph neural network (GNN) architecture specifically designed to process graph data with multiple relationship types. The design purpose of RGCN is to be able to effectively capture the complex relationships between different types of nodes and edges in the graph. The concatenated feature z is input into the relational graph neural network to extract the node feature, as shown in formula (13).

[0126]

[0127] Among them, Θ root , Θ r are the learned parameters, |N r (i)| is the normalization constant, z i and z j are the i-th and j-th features of the concatenated feature z; R represents the set of relationships.

[0128] For the node feature x′ i , x′ j (i,j∈1,2,...n), perform the operation shown in formula (14) to obtain the fused feature.

[0129] H′ i =W 1 x′ i +∑α i,j W 2 x′ j (14)

[0130] In the formula, is the attention coefficient; W 1 , W 2 , W3 and W 4 are both learnable weight matrix parameters; x' i and x' j represent two node features obtained from the heterogeneous graph neural network; m represents the dimension of the feature space, and softmax represents the activation function.

[0131] Then, in S5, the fused features are input into the classification model for training.

[0132] According to the embodiments of the present invention, the multi-modal sentiment classification model can be classified using models such as a fully connected neural network (FCN), a Transformer model, and an attention mechanism. In the embodiments of the present invention, a fully connected neural network (FCN) is used for sentiment classification.

[0133] First, the fused features are input into a fully connected layer, as shown in formula (15); then, they are converted into an output representing a probability distribution using the softmax function, as shown in formula (16); finally, the maximum probability is taken as the prediction label, as shown in formula (17). The obtained is the sentiment label predicted by the model. Formulas (18)-(19) are the explanatory notes for the softmax function and the ReLU function.

[0134] H i = ReLU(W 4 H′ i + b 1 ) (15)

[0135] P i = softmax(W 5 H i + b 2 ) (16)

[0136]

[0137] Finally, in S6, the data to be tested is input into the trained classification model for classification to obtain the sentiment recognition result.

[0138] According to the embodiments of the present invention, the data to be tested are the text features, speech features, and image features after feature extraction.

[0139] The technical effects of the present invention are further verified through experiments.

[0140] The datasets used in the experiments are IEMOCAP, CMU-MOSEI, and MELD, all of which are publicly available multimodal datasets. Among them, IEMOCAP contains two conversation videos of 12 hours from 10 speakers. Each conversation is divided into several utterances. There are a total of 7433 utterances and 151 conversations. The 6-class dataset contains 6 emotion labels, namely happy, sad, neutral, angry, excited, and frustrated, which are assigned to the utterances. As a simplified version, ambiguous data pairs such as (happy, withdrawn) and (happy, frustrated) are merged into a 4-class dataset. CMU-MOSEI is a dataset for multimodal emotion recognition. It contains annotations of 7 emotions, ranging from highly negative (-3) to highly positive (+3), and 6 emotion labels, including happy, sad, disgusted, fearful, surprised, and angry. MELD is a multimodal, multi-speaker conversation dataset that has three high-quality modality-aligned conversation data, with a total of 13708 utterances, 1433 conversations, and 304 different speakers. Specifically, different from binary conversation datasets such as IEMOCAP, MELD has three or more speakers in a conversation. Each utterance in the conversation is annotated with 7 emotion labels, including angry, disgusted, fearful, joyous, neutral, sad, and surprised.

[0141] The weighted F1 score and accuracy are used as evaluation metrics. Among them, the F1 score is the harmonic mean of precision and recall. When either precision or recall decreases, the F1 value will decrease; conversely, the F1 value will increase. The calculation method is shown in formula (20).

[0142]

[0143] Accuracy is the percentage of correct predictions in the test set.

[0144] The experiments compared the present invention with models in recent years, and the results showed that the performance of the model proposed by the present invention has been significantly improved. Tables 1-3 give the comparison results.

[0145] Table 1 Comparison Experiment Results of IEMOCAP Dataset

[0146]

[0147]

[0148] Table 2 Comparison Experiment Results of CMU-MOSEI Dataset

[0149]

[0150] Table 3 Comparison Experiment Results of MELD Dataset

[0151]

[0152] On the IEMOCAP dataset, except for the Neutral and Excited categories, the model proposed by the present invention outperforms the baseline model in other categories. This means that the model has higher accuracy and weighted F1 values when identifying emotional categories other than neutral and excited. The accuracy has increased by 2.01%, indicating that the model has made significant improvements in correctly classifying samples. The weighted F1 value has increased by 2.37%, indicating that the model has achieved better results when balancing accuracy and recall.

[0153] On the CMU-MOSEI dataset, except for the Happy category, the results of the model proposed by the present invention are also better than those of the baseline model in other categories. Whether in the binary classification task or the seven-class classification task, the accuracy of this model is at least 0.71% and 1.09% higher than that of other comparison models. This means that the model can more accurately classify emotional categories other than happy, whether in binary classification or multi-class classification.

[0154] On the MELD dataset, the model proposed by the present invention has at least 0.94% and 1.31% higher accuracy and weighted F1 values respectively than other comparison models. This shows that the model has improved in the accuracy of emotion recognition and has achieved better overall performance when balancing accuracy and recall. These results further verify the superiority of the model proposed by the present invention in multi-modal emotion recognition tasks.

[0155] In summary, the method proposed by the present invention shows better performance than the baseline model on the three datasets of IEMOCAP, CMU-MOSEI, and MELD. It not only achieves better accuracy and weighted F1 values in categories other than specific emotional categories, but also shows better performance in multi-modal emotion recognition tasks.

[0156] Further ablation experiments were conducted to verify the effectiveness of each module.

[0157] First, consider the influence of modalities. Table 4 shows the results of the TL-RGCN model on the IEMOCAP dataset obtained from different modality combinations.

[0158] Table 4 Performance of the model under different modality settings

[0159]

[0160] It can be seen that for the IEMOCAP dataset, the text modality performs best in the unimodal setting, while the visual modality yields the lowest results. This may be due to the presence of noise, such as camera position, environmental conditions, etc. In the bimodal setting, combining the text and audio modalities achieves the best performance, while combining the audio and visual modalities gives the worst results. In addition, the combination of the three modalities yields the optimal effect.

[0161] Secondly, consider the importance of each module. The role of the domain generalization module is to further process the features obtained by the feature extraction module, making them better adapted to subsequent feature fusion. The RGCN (Relational Graph Convolutional Network) module can better fuse the features of the three modalities and reflect the relationship R within the same speaker intra and the relationship R between different speakers inter . Therefore, the present invention conducts ablation experiments to verify its effectiveness. The results are shown in Table 5.

[0162] Table 5 Performance of the model after ablation of different modules

[0163]

[0164] As can be seen from Table 5, the RGCN module has the greatest impact on the model. If removed, the performance of the model drops by 3.80% (w-F1). At the same time, TL (domain generalization module) is also a key part of the model and will also have a greater impact on the model performance, causing its w-F1 value to drop by 3.19%. intra and R inter are both part of the RGCN module, and the two complement each other and have a certain impact on the model performance.

[0165] The present invention plots the confusion matrix for the classification results of the model on the IEMOCAP dataset, as Figure 4 shown. The confusion matrix provides an overview of the comprehensive performance of the model in emotion classification and allows for a detailed assessment of the classification accuracy of each emotion category. It can be seen that the model proposed by the present invention has almost no errors in predicting emotions of other dissimilar categories. This indicates that the model proposed by the present invention has relatively good performance in solving the discrimination problem between certain emotions.

[0166] The present invention proposes a new model TL-RGCN, which includes two key modules: a domain generalization module and a graph neural network module. The former performs cross-modal interaction by taking the features extracted from a single modality as a domain respectively, so that the extracted features are more suitable for subsequent multi-modal feature fusion. The latter learns global and local information through the graph neural network to capture the emotions of the interlocutors. Experimental results on three datasets, namely IEMOCAP, CMU-MOSEI, and MELD, show that the combination of the two modules can greatly improve the performance of multi-modal emotion recognition.

[0167] Another embodiment of the present invention proposes a multi-modal emotion recognition system based on domain generalization and graph neural network, as Figure 5 shown. The system includes:

[0168] A data acquisition module 510 configured to acquire a dataset, where the dataset includes three types of modal data: text modality, speech modality, and visual modality;

[0169] A feature extraction module 520 configured to extract features from the three types of modal data to obtain text features, speech features, and image features;

[0170] A domain generalization module 530 configured to perform domain generalization with each modality as a domain, that is, divide the features extracted from each modality into two parts, one part is used to extract intra-domain invariant features, and the other part is used to extract inter-domain invariant features;

[0171] A feature fusion module 540 configured to fuse the intra-domain invariant features and the inter-domain invariant features using a graph neural network to simultaneously obtain global information and local context information;

[0172] A model training module 550 configured to input the fused features into a classification model for training;

[0173] An emotion recognition module 560 configured to input the data to be measured into the trained classification model for classification to obtain an emotion recognition result.

[0174] The functions of the multi-modal emotion recognition system based on domain generalization and graph neural network proposed in the embodiments of the present invention can be illustrated by the foregoing multi-modal emotion recognition method based on domain generalization and graph neural network. Therefore, for the parts not detailed in the system embodiments, reference can be made to the above method embodiments, which will not be elaborated here.

[0175] It should be noted that although several units, modules or sub-modules are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0176] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0177] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefit. This division is only for the convenience of expression. The present invention aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A multimodal emotion recognition method based on domain generalization and graph neural network, characterized by: include: Acquire a data set, wherein the data set includes three modal data: text modality, voice modality, and visual modality; Performing feature extraction on the three modal data to obtain text features, voice features and image features; Each modality is treated as a domain for domain generalization, that is, the features extracted by each modality are divided into two parts, one part is used to extract invariant features within the domain, and the other part is used to extract invariant features between domains; Using a graph neural network to fuse the intra-domain invariant features and the inter-domain invariant features, so as to simultaneously obtain global information and local context information; Input the fused features into the classification model for training; The test data is input into the trained classification model for classification to obtain the emotion recognition results.

2. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 1 is characterized in that: For text modality, the RoBERTa model is used for pre-training and fine-tuning to extract text features; for speech modality, the OpenSmile tool is used to extract speech features; For the visual modality, a DenseNet model is used for feature extraction of facial images.

3. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 1 is characterized in that: The domain generalization of each modality as a domain includes: A distillation algorithm is used to learn domain-invariant features, including: the teacher network uses Fourier phase information and class labels as input and output, respectively, to obtain Fourier phase information features for classification; The CORAL alignment algorithm is used to learn inter-domain invariant features; The objective function for determining domain generalization is as follows: In the formula, The loss function corresponding to the invariant features in the learning domain; The loss function corresponding to learning the invariant features between domains; represents the loss function corresponding to the introduced regular term; θ f ,θ c are the parameters to be solved, representing the feature extraction network G f and classifier G c The learnable parameters of Represents the distribution of training feature data p tr expectations; represents the cross entropy loss; represents the mean square error loss; represents the true value of the teacher network; (x, y) represents the training feature data, x represents the feature data, and y is the corresponding label; represents the student network feature extractor; z2=G f (x); λ1, λ2, λ3 are hyperparameters; min means minimization; The above objective function is iteratively solved using the optimization algorithm to update the model parameters θ f ,θ c , to optimize the intra-domain invariant features z1 and inter-domain invariant features z2.

4. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 3 is characterized in that: The loss function of learning domain-invariant features for: In the formula, C i , C j is the covariance matrix composed of sample features from different fields; ||·|| F is the norm; N represents the number of feature samples; i and j represent sample indexes between different fields; The loss function corresponding to the introduced regularization term is for: Where d(·,·) is the distance function.

5. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 4 is characterized in that: The fusing of the intra-domain invariant features and the inter-domain invariant features by using a graph neural network includes: Connect the extracted invariant features within the domain and the invariant features between domains to obtain the connected features Input the concatenated feature z into the heterogeneous graph neural network to extract node features; For different node features, the attention mechanism is used to fuse them and obtain the fused features.

6. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 5 is characterized in that: The connected feature z is input into the heterogeneous graph neural network, and the formula for extracting node features is: In the formula, Θ root , Θ r is the learning parameter; |N r (i) | is a normalization constant; z i and z j are the i-th and j-th features of feature z after concatenation; R represents a set of relations.

7. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 6 is characterized in that: For different node features, the formula for fusing using the attention mechanism is: H′ i =W1x′ i +∑α i,j W2x′ j In the formula, is the attention coefficient; W1, W2, W3 and W4 are all learnable weight matrix parameters; x' i 、x' j Represents two node features obtained from the heterogeneous graph neural network; m represents the dimension of the feature space, and softmax represents the activation function.

8. The multimodal emotion recognition method based on domain generalization and graph neural network according to claim 7 is characterized in that: The step of inputting the fused features into a classification model for training includes: the classification model adopts a fully connected neural network; inputting the fused features into a fully connected layer; using a softmax activation function to convert the output of the fully connected layer into a probability distribution, and taking the maximum value in the probability distribution as the predicted sentiment label.

9. The multimodal emotion recognition method based on domain generalization and graph neural network according to any one of claims 1 to 8, characterized in that: The emotion recognition results include: happiness, sadness, neutral, anger, excitement, frustration, disgust, fear, and surprise.

10. A multimodal emotion recognition system based on domain generalization and graph neural network, characterized by: include: A data acquisition module configured to acquire a data set, wherein the data set includes three modality data: text modality, voice modality, and visual modality; A feature extraction module configured to extract features from the three modal data to obtain text features, speech features, and image features; A domain generalization module is configured to treat each modality as a domain for domain generalization, that is, to divide the features extracted by each modality into two parts, one part is used to extract invariant features within the domain, and the other part is used to extract invariant features between domains; A feature fusion module, configured to fuse the intra-domain invariant features and the inter-domain invariant features using a graph neural network to simultaneously obtain global information and local context information; A model training module, configured to input the fused features into a classification model for training; The emotion recognition module is configured to input the test data into the trained classification model for classification and obtain the emotion recognition result.

Images