A multimodal conversation emotion recognition method

Abstract

The present invention proposes a multimodal conversation emotion recognition method, which belongs to the technical field of conversation emotion recognition. It includes: S1. Using different encoders to encode the unimodal data of each round of conversation to obtain the feature representation of each modality in the conversation; S2. Aggregating the contextual conversation information of a round of conversation; S3. Performing linear transformation on the multimodal features of each sentence interaction in the conversation and then fusing the features between the modalities; S4. Using an independent classifier to classify the features of each modality based on the auxiliary task, using the main task classifier to perform main task emotion classification on the fused features, calculating the classification loss separately, and the final model loss is weighted by the auxiliary task and main task losses; S5. Inputting the multimodal data into the main task classifier and outputting the emotion category. This method solves the problem that the emotion information is not fully mined in combination with all conversation contexts, and the correlation of the emotions expressed by each modal data is not fully utilized.

Description

Technical Field

[0001] The present application relates to a method for recognizing emotions in conversations, and in particular to a method for recognizing emotions in multimodal conversations, belonging to the technical field of conversation emotion recognition. Background Art

[0002] With the rapid development of human-computer interaction and intelligent customer service, emotion recognition, as a key technology, has garnered widespread attention and application. Today, large-scale social media platforms continuously generate vast amounts of conversational data, including but not limited to text, audio, and video. Studying the emotional tendencies of speakers has significant application value in areas such as public opinion analysis, e-customer service, and healthcare. Consequently, research on emotion recognition in multimodal conversations is gaining increasing attention from researchers and businesses. Unlike traditional single-modal research focused on isolated utterances, multimodal conversational emotion recognition not only integrates conversational context to uncover underlying emotions, but also simultaneously considers multiple modal information within a conversation, such as speech, text, and images, to more accurately capture user emotions and needs.

[0003] Previous research has proposed numerous models for capturing conversational context. CMN employs a multimodal approach, incorporating acoustic, visual, and textual features with gated recurrent units (GRUs) to model each speaker's past utterances. DialogueRNN uses different GRU units to capture contextual information and speaker relationships. It is the first speaker-differentiated conversational sentiment analysis model. DialogueGCN constructs a conversation as a graph, transforming the speech sentiment classification problem into a graph node classification problem and classifying the results using a graph convolutional neural network. DialogueCRN introduces a cognitive phase, extracting and integrating emotional cues from the context retrieved during the perception phase for context modeling. Some of these models focus solely on the textual modality of the conversation, ignoring the contributions of other modalities, while others fail to model speaker information, thereby losing some conversational context. MMGCN utilizes a GCN network to capture contextual information, effectively leveraging both multimodal dependencies and speaker information. However, MMGCN combines data from different modalities into a single graph for training, failing to consider the impact of individual modal differences on the overall model performance. Summary of the Invention

[0004] A brief overview of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important aspects of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0005] In view of this, in order to solve the technical problems in the existing technology that most emotion recognition methods only consider the contextual information within a limited window, but do not fully mine the emotion information in combination with all the dialogue contexts, and the unity and correlation of the emotions expressed by each modal data are not fully utilized, the present invention provides a multimodal dialogue emotion recognition method. In order to fully aggregate the dialogue context information and maximize the contribution of each modality to the final emotion recognition effect, the present invention proposes a new multimodal dialogue emotion recognition method based on graph neural network and multi-task learning. First, the single modal dialogue context is modeled, and the speaker feature embedding is added at the same time. The aggregated features are respectively sent to the single modal auxiliary task classification part and the multimodal feature fusion part, and the fused features are sent to the final multimodal emotion classifier main task for classification, and the loss function is designed by multi-task learning to jointly optimize the main task and auxiliary task.

[0006] Solution 1: A multimodal conversation emotion recognition method, comprising the following steps:

[0007] S1. Use different encoders to encode the unimodal data of each round of dialogue to obtain the feature representation of each modality in the dialogue, including the following steps:

[0008] S11. Use TextCNN to extract the original features of sentence text;

[0009] S12. Use a bidirectional long short-term memory network to preliminarily model sentence text sequences, encode long-term dependencies between sentence text sequences, and learn contextual information in sentence sequences.

[0010]

[0011] in, Represents the original feature representation that is independent of the text modality context, Indicates that the previous sentence is a high-level feature representation from the text modality. Indicates that the current sentence comes from the high-level feature representation of the text modality, Indicates that the next sentence comes from a high-level feature representation of the text modality;

[0012] S13.OpenSmile toolkit extracts raw acoustic features;

[0013] S14. Use the DenseNet model pre-trained on the FER+ corpus to extract raw features of facial expressions;

[0014] S15. Use a fully connected network to model the raw features of speech and visual modalities;

[0015]

[0016]

[0017] in, represents high-level feature representations from the speech modality, represents the weight parameter of training, represents the original feature representation of the acoustic modality that is independent of the context, represents the bias parameter, represents a high-level feature representation of the visual modality, represents the weight parameter of training, represents the bias parameter, Representing raw feature representations that are context-independent of the visual modality;

[0018] S16. Encode the speaker's features and embed them into the original features. The original speaker features are represented as a one-hot vector s i , the embedding S of speaker features i The calculation method is as shown in the formula:

[0019]

[0020] Among them, W k is a trainable weight parameter, s k One-hot vector representing speaker features, b k is the bias parameter;

[0021] The original features embedded with speaker information are obtained as shown in the following formula:

[0022]

[0023]

[0024]

[0025] in represents the original features embedded with speaker information, is the original feature without speaker feature embedding, S k Represents the speaker embedding feature of the current sentence;

[0026] S2. Aggregate the contextual dialogue information of a round of dialogue using a deep graph convolutional network;

[0027] S3. Perform linear transformation on the text, speech, and visual features of each sentence in the conversation and then perform feature fusion between the modalities.

[0028] S4. Based on the auxiliary task, use an independent classifier to classify the features of each modality. Use the main task classifier to perform main task sentiment classification on the fused features. Calculate the classification loss for each modality. The final model loss is calculated by weighting the auxiliary task and main task losses.

[0029] S5. Input the data of text modality, speech modality and visual modality into the main task classifier and output the emotion category.

[0030] Preferably, the method for aggregating the contextual dialogue information of a dialogue round using a deep graph convolutional network comprises the following steps:

[0031] S21. Suppose there is a conversation with N utterances. The data of text mode, speech mode and visual mode of each utterance are represented as

[0032] S22. Use graph neural networks to construct three undirected graphs G for the text mode, voice mode, and visual mode of each sentence t =(V t ,E t ), G a =(V a ,E a ),G v =(V v ,E v ); among them, G t , G a , G v They represent the text modality undirected graph, the speech modality undirected graph, and the visual modality undirected graph, respectively. V represents the set of discourse nodes, and E represents the set of adjacency relationships between the feature nodes of each sentence.

[0033] S23. Use a graph neural network to construct a graph for all text, speech, and visual modalities of all utterances, and for all sample data of the same modality, connect each sample node to every other sample node.

[0034] S24. Calculate the similarity A between sample nodes based on the Pearson correlation coefficient xy :

[0035]

[0036] Among them, x i and y i Respectively represent the values ​​of the two samples at the i-th feature, and Represents the mean of all features of the two samples respectively;

[0037] S25. Add the initial residual connection and identity mapping to the GCN network to aggregate context features. Let is the graph convolution matrix of the undirected graph G:

[0038]

[0039] Among them, A represents the adjacency matrix, D represents the diagonal matrix of the graph G, and I represents the identity matrix; the operation between the multi-layer graph convolutional network is shown as follows:

[0040]

[0041] Among them, α and β (l) Represent two hyperparameters, σ represents the activation function, W (l) represents the learnable weight matrix;

[0042] set up Where η represents a hyperparameter, the first layer H (0) The residual connection is added to the representation of each layer and use the unit map I plus the weight matrix W (l) , and obtain the feature representation of the current layer that aggregates the information of the previous layer.

[0043] Preferably, the method for performing linear transformation on the text modality, speech modality, and visual modality features after the interaction of each sentence in the conversation and then fusing the features between the modalities is:

[0044]

[0045]

[0046]

[0047] F i m =concat[F i t ,F i a ,F i v ]

[0048] in, and Respectively represent the text modality, acoustic modality and visual modality aggregated conversation context information features, F i t , F i a , F i v Respectively and After linear transformation, the feature W t 、W a and W vis the training weight parameter, b t 、b a and b v is the bias parameter, F i m It represents the fusion features of text modality, speech modality and visual modality, and concat[·] represents the concatenation operation.

[0049] Preferably, the classifier calculation process is as shown in the formula:

[0050] F i ' k =σ(W k F i k +b k )

[0051]

[0052]

[0053] Among them, k∈{t,a,v,m}, F i k represents a single modal feature or fusion feature entering the classifier, σ(·) represents the activation function, F i ' k Represents the features after the activation function calculation, Softmax(·) normalizes the input features into probability distribution, P i k represents the probability that the sample belongs to each category, W k and represents the weight parameter of training, b k and represents the bias parameter, represents the classification result, argmax(·) calculates the category with the highest classification probability, and c represents the number of categories in all samples.

[0054] Preferably, the classification loss function is calculated as shown in the formula:

[0055]

[0056] Where N is the number of dialogues, c(i) is the number of utterances in the dialogue, k∈{t,a,v,m}; α k represents the weight of each subtask loss, represents the probability distribution of emotion labels predicted by utterance j in conversation i in subtask k, y i,j represents the expected class label of utterance j in dialogue i, λ represents the L2 regularization weight, and θ represents the set of all training parameters.

[0057] Solution 2: An electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multimodal dialogue emotion recognition method described in Solution 1.

[0058] Solution 3: A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal dialogue emotion recognition method described in Solution 1.

[0059] The present invention has the following beneficial effects: It effectively extracts contextual information from each modality and, through multi-task learning and feature fusion classification, improves the performance of emotion recognition tasks. Compared to many existing emotion classification models, the present invention achieves varying degrees of performance improvement, demonstrating its effectiveness and superiority. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 A flowchart of a multimodal conversation emotion recognition method. DETAILED DESCRIPTION

[0062] In order to make the technical solutions and advantages of the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, and are not an exhaustive list of all the embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other unless they conflict.

[0063] Example 1, reference Figure 1 This embodiment describes a multimodal conversation emotion recognition method, comprising the following steps:

[0064] Sufficient original feature extraction is performed on the data of each modality. The more emotion-related features extracted from a single modality, the richer the features extracted later. In addition, the quality of single-modal features also affects the effect of information interaction within and between modalities during multimodal feature fusion.

[0065] S1. Use different encoders to encode the unimodal data of each round of dialogue to obtain feature representations of each mode of dialogue;

[0066] S11. Use TextCNN to extract the original features of sentence text;

[0067] S12. Use a bidirectional long short-term memory network to preliminarily model sentence text sequences, encode long-term dependencies between sentence text sequences, and learn contextual information in sentence sequences.

[0068]

[0069] in, Represents the original feature representation that is independent of the text modality context, Indicates that the previous sentence is a high-level feature representation from the text modality. Indicates that the current sentence comes from the high-level feature representation of the text modality, Indicates that the next sentence comes from a high-level feature representation of the text modality;

[0070] S13.OpenSmile toolkit extracts raw acoustic features;

[0071] S14. Use the DenseNet model pre-trained on the FER+ corpus to extract raw features of facial expressions;

[0072] S15. Use a fully connected network to model the raw features of speech and visual modalities;

[0073]

[0074]

[0075] in, represents high-level feature representations from the speech modality, represents the weight parameter of training, represents the original feature representation of the acoustic modality that is independent of the context, represents the bias parameter, represents a high-level feature representation of the visual modality, represents the weight parameter of training, represents the bias parameter, Representing raw feature representations that are context-independent of the visual modality;

[0076] S16. Encode the speaker's features and embed them into the original features. The original speaker features are represented as a one-hot vector s i , the embedding S of speaker features i The calculation method is as shown in the formula:

[0077]

[0078] Among them, W k is a trainable weight parameter, s k One-hot vector representing speaker features, b kis the bias parameter;

[0079] The original features embedded with speaker information are obtained as shown in the following formula:

[0080]

[0081]

[0082]

[0083] in represents the original features embedded with speaker information, is the original feature without speaker feature embedding, S k Represents the speaker embedding feature of the current sentence.

[0084] In the task of conversational emotion recognition, unlike the task of single-sentence emotion recognition, the previous and subsequent conversations in a round of conversation will contain information related to the current round of conversation. In order to further extract the contextual features at the dialogue level of each modality, a deep graph convolutional network is used to aggregate all conversation features of the same modality.

[0085] S2. Aggregate the contextual dialogue information of a round of dialogue using a deep graph convolutional network;

[0086] S21. Suppose there is a conversation with N utterances. The data of text mode, speech mode and visual mode of each utterance are represented as

[0087] S22. Use graph neural networks to construct three undirected graphs G for the text mode, voice mode, and visual mode of each sentence t =(V t ,E t ), G a =(V a ,E a ),G v =(V v ,E v ); where G t , G a , G v They represent the text modality undirected graph, the speech modality undirected graph, and the visual modality undirected graph, respectively. V represents the set of discourse nodes, and E represents the set of adjacency relationships between the feature nodes of each sentence.

[0088] S23. Use a graph neural network to construct a graph for all text, speech, and visual modalities of all utterances, and for all sample data of the same modality, connect each sample node to every other sample node.

[0089] If the correlation between two nodes is higher, more important information can be obtained from each other during information exchange, so the weight of the edge connecting two highly correlated nodes is also greater;

[0090] S24. Calculate the similarity A between sample nodes based on the Pearson correlation coefficient xy :

[0091]

[0092] Among them, x i and y i Respectively represent the values ​​of the two samples at the i-th feature, and Represents the mean of all features of the two samples respectively;

[0093] Get the feature vectors of nodes in a graph and the definition of the relationship edges between nodes. Construct graph structures for the text modality, speech modality, and visual modality features of a conversation process. The following formula is the representation of text modality:

[0094]

[0095]

[0096] G t =(H t ,A t )

[0097] in, H represents the feature vector of all utterances in the text modality of a paragraph that are embedded with speaker features, t Represents the set of the above feature vectors, that is, the set of nodes in the graph structure, A t Represents the similarity matrix between the features of each sentence, that is, the adjacency matrix of the nodes in the graph structure, G t A graph representing the modal features of all utterances in a passage;

[0098] S25. Add the initial residual connection and identity mapping to the GCN network to aggregate context features. Let is the graph convolution matrix of the undirected graph G:

[0099]

[0100] Among them, A represents the adjacency matrix, D represents the diagonal matrix of the graph G, and I represents the identity matrix; the operation between the multi-layer graph convolutional network is shown as follows:

[0101]

[0102] Among them, α and β (l)Represent two hyperparameters, σ represents the activation function, W (l) represents the learnable weight matrix;

[0103] set up Where η represents a hyperparameter, the first layer H (0) The residual connection is added to the representation of each layer and use the unit map I plus the weight matrix W (l) , get the feature representation of the current layer that aggregates the information of the previous layer;

[0104] After the speech features in the conversation interact in the deep graph convolutional network of each modality, they already have the conversation context information of each modality. The trimodal features after each sentence interaction are linearly transformed and then the features between the modalities are fused.

[0105] S3. Perform linear transformation on the text, speech, and visual features of each sentence in the conversation and then perform feature fusion between the modalities.

[0106]

[0107]

[0108]

[0109] F i m =concat[F i t ,F i a ,F i v ]

[0110] in, and Respectively represent the text modality, acoustic modality and visual modality aggregated conversation context information features, F i t , F i a , F i v Respectively and After linear transformation, the feature W t 、W a and W v is the training weight parameter, b t 、b a and b v is the bias parameter, F i mIt represents the fusion features of text modality, speech modality and visual modality, and concat[·] represents the concatenation operation.

[0111] Direct concatenation retains all feature information without loss or change, and is not restricted by feature dimension or type, making it suitable for multimodal dialogue emotion recognition tasks. In terms of interpretability, direct concatenation does not perform any processing on the input features. The concatenated feature vector is a combination of the original feature vectors, making the model prediction results better understood and explained.

[0112] S4. Based on the auxiliary task, use an independent classifier to classify the features of each modality, use the main task classifier to perform main task sentiment classification on the fused features, calculate the classification loss separately, and the final model loss is weighted by the auxiliary task and main task losses.

[0113] The classifier calculation process is shown in the formula:

[0114] F i ' k =σ(W k F i k +b k )

[0115]

[0116]

[0117] Among them, k∈{t,a,v,m}, F i k represents a single modal feature or fusion feature entering the classifier, σ(·) represents the activation function, F i ' k Represents the features after the activation function calculation, Softmax(·) normalizes the input features into probability distribution, P i k represents the probability that the sample belongs to each category, W k and represents the weight parameter of training, b k and represents the bias parameter, represents the classification result, argmax(·) calculates the category with the highest classification probability, and c represents the number of categories in all samples;

[0118] The classification loss function is calculated as shown in the formula:

[0119]

[0120] Where N is the number of dialogues, c(i) is the number of utterances in the dialogue, k∈{t,a,v,m}; α k represents the weight of each subtask loss, represents the probability distribution of emotion labels predicted by utterance j in conversation i in subtask k, y i,j represents the expected class label of utterance j in dialogue i, λ represents the L2 regularization weight, and θ represents the set of all training parameters.

[0121] S5. Input the data of text modality, speech modality and visual modality into the main task classifier and output the emotion category.

[0122] Experimental evaluation of the present invention is conducted on two benchmark datasets: IEMOCAP and MELD.

[0123] IEMOCAP and MELD are multimodal datasets that contain acoustic, visual, and textual information for each utterance in a conversation. Both datasets are split into training and test sets in a ratio of approximately 4:1. Table 1 (Dataset Sample Distribution Table) shows the distribution of training and test samples for these two datasets.

[0124] Table 1 Dataset sample distribution table

[0125]

[0126] IEMOCAP dataset: Contains 12 hours of two-way conversation videos from 10 unique speakers, with only the first 8 speakers from sessions 1 to 4 used in the training set; the video contains a single binary conversation, split into utterances; there are a total of 7433 utterances and 151 conversations; each sentence in the conversation is annotated with six categories of emotion labels, including happy, sad, neutral, angry, excited, and frustrated.

[0127] The MELD dataset is a multimodal, multi-speaker conversation dataset with high-quality three-modal aligned conversation data. It contains 13,708 utterances, 1,433 conversations, and 304 different speakers. Unlike binary conversation datasets such as IEMOCAP, MELD has three or more speakers in a conversation. Each sentence in the conversation is annotated with seven emotion labels, including anger, disgust, fear, joy, neutrality, sadness, and surprise.

[0128] The experiment uses Python 3.7.13 and the deep learning framework Pytorch 1.8.0. The experimental environment runs on a Windows 10 system. The experimental hardware is an Intel (R) Xeon (R) W-2123 processor and a GeForce GTX 1080Ti graphics card. Table 2 shows the experimental parameter settings.

[0129] Table 2 Experimental parameter settings

[0130]

[0131] It's worth noting that the AdamW optimizer was used to optimize the model. Compared to the Adam optimizer, the AdamW optimizer can better control the model's weight decay, providing a more stable training process and better performance. This has been proven effective.

[0132] The evaluation metrics used for classification tasks are accuracy, precision, and recall; accuracy is defined as the proportion of correctly classified samples to the total number of samples, which reflects the credibility of the predictions made by the model.

[0133] The F1 value is the harmonic mean of precision and recall. If any of the precision and recall values ​​decrease, the F1 value will decrease, and vice versa. The weighted average F1 value is used to evaluate the results. The calculation method of the weighted average F1 value is shown in the following formula:

[0134]

[0135]

[0136] Among them, P (Precision) represents the ratio of the number of samples correctly predicted as positive examples by the classifier to the total number of samples predicted as positive examples, R (Recall) represents the ratio of the number of samples correctly predicted as positive examples by the classifier to the total number of true positive examples, and F1 i represents the F1 value of category i, w i represents the weight of category i.

[0137] To verify the effectiveness of the proposed model in the task of multimodal emotion recognition in conversations, its performance is compared with the following methods:

[0138] BC-LSTM: BC-LSTM uses a bidirectional LSTM structure to encode contextual semantic information and does not identify speaker relationships.

[0139] CMN: adopts a multimodal approach that incorporates audio, visual, and textual features, with gated recurrent units to model each speaker's past utterances.

[0140] DialogueRNN: DialogueRNN uses different GRU units to obtain contextual information and speaker relationships. It is the first conversational sentiment analysis model to distinguish speakers.

[0141] DialogueGCN: DialogueGCN constructs the conversation as a graph, converts the speech emotion classification problem into a node classification problem of the graph, and uses a graph convolutional neural network to classify the results.

[0142] DialogueCRN: DialogueCRN introduces a cognitive stage to extract and integrate emotional cues from the context retrieved in the perception stage for context modeling.

[0143] MMGCN: MMGCN uses the GCN network to obtain contextual information, which can effectively utilize multimodal dependencies and speaker information.

[0144] The experimental results are analyzed with reference to the effect table of each model on IEMOCAP and MELD in Table 3; Table 3 shows the effect of the MulTDGCN model on the IEMOCAP and MELD datasets; the MulTDGCN (present invention) model performs better than the previous baseline models on both datasets. Among them, the DialogueGCN model is a model for emotion recognition in a single text modality, and has better recognition performance for the text modality. The MMGCN model adds acoustic and visual modalities, which has a certain improvement over the single-modality DialogueGCN model. The MulTDGCN model is also an emotion recognition model that uses multimodal data. Compared with the MMGCN model, it improves the F1 value score by 1.83% and 0.11% on the IEMOCAP and MELD datasets, respectively.

[0145] Table 3 Effects of each model on IEMOCAP and MELD

[0146]

[0147] Table 4 (Model Performance under Different Modality Settings) compares the performance of MulTDGCN on the two datasets under different settings for the modalities involved in training. As can be seen from Table 4, when only a single modality is involved in training, the text modality performs best, and the visual modality performs worst. Adding the text modality to the acoustic or visual modality for training achieves better results than either modality alone. The model achieves the best results when all three modalities are used for training.

[0148] Table 4 Performance of the model under different modal settings

[0149]

[0150] We explore whether to add auxiliary tasks to the model. From Table 5 (Table 5 Effect of auxiliary tasks on model performance

[0151] ) It can be seen that the performance of the model on both datasets has been improved after adding auxiliary tasks.

[0152] Table 5. Impact of auxiliary tasks on model performance

[0153]

[0154] We studied the impact of the number of graph convolutional network layers on model performance. Table 6 shows that increasing the number of graph convolutional network layers improves model performance, but after reaching a certain number of layers, performance gradually declines. The ideal number of layers was determined to be 64 on the IEMOCAP and 8 on the MELD datasets.

[0155] Table 6 The impact of different graph neural network layers on model performance

[0156]

[0157] In summary, the results demonstrate that MulTDGCN outperforms previous methods in multimodal conversational emotion recognition. Analysis and ablation experiments on MulTDGCN demonstrate the importance of each module. MulTDGCN effectively extracts contextual information from each modality and optimizes features through multi-task learning for fusion classification, improving performance in emotion recognition tasks.

[0158] In Example 2, the computer device of the present invention may include a processor and memory, such as a single-chip microcomputer including a central processing unit. Furthermore, the processor is configured to execute a computer program stored in the memory to implement the steps of the aforementioned multimodal conversation emotion recognition method.

[0159] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0160] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); and the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0161] Example 3: Computer-readable storage medium example

[0162] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned multimodal dialogue emotion recognition method can be implemented.

[0163] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0164] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A multimodal dialogue emotion recognition method, characterized in that: The following steps are involved: S1. Use different encoders to encode the unimodal data of each round of dialogue to obtain the feature representation of each modality in the dialogue, including the following steps: S11. Use TextCNN to extract the original features of sentence text; S12. Use a bidirectional long short-term memory network to preliminarily model sentence text sequences, encode long-term dependencies between sentence text sequences, and learn contextual information in sentence sequences. in, Represents the original feature representation that is independent of the text modality context, Indicates that the previous sentence is a high-level feature representation from the text modality. Indicates that the current sentence comes from the high-level feature representation of the text modality, Indicates that the next sentence comes from a high-level feature representation of the text modality; S13.OpenSmile toolkit extracts raw acoustic features; S14. Use the DenseNet model pre-trained on the FER+ corpus to extract raw features of facial expressions; S15. Use a fully connected network to model the raw features of speech and visual modalities; in, represents high-level feature representations from the speech modality, represents the weight parameter of training, represents the original feature representation of the acoustic modality that is independent of the context, represents the bias parameter, represents a high-level feature representation of the visual modality, represents the weight parameter of training, represents the bias parameter, Representing raw feature representations that are context-independent of the visual modality; S16. Encode the speaker's features and embed them into the original features. The original speaker features are represented as a one-hot vector s i , the embedding S of speaker features i The calculation method is as shown in the formula: Among them, W s is a trainable weight parameter, s i A one-hot vector representing speaker characteristics, is the bias parameter; The original features embedded with speaker information are obtained as shown in the following formula: in represents the original features embedded with speaker information, is the original feature without speaker feature embedding, S k Represents the speaker embedding feature of the current sentence; S2. Aggregate the contextual dialogue information of a round of dialogue using a deep graph convolutional network; S3. Perform linear transformation on the text, speech, and visual features of each sentence in the conversation and then perform feature fusion between the modalities. S4. Based on the auxiliary task, use an independent classifier to classify the features of each modality. Use the main task classifier to perform main task sentiment classification on the fused features. Calculate the classification loss for each modality. The final model loss is calculated by weighting the auxiliary task and main task losses. S5. Input the data of text modality, speech modality and visual modality into the main task classifier and output the emotion category.

2. A multimodal dialogue emotion recognition method according to claim 1, characterized in that: The method of aggregating the contextual dialogue information of a round of dialogue using a deep graph convolutional network includes the following steps: S21. Suppose there is a conversation with N utterances. The data of text mode, speech mode and visual mode of each utterance are represented as S22. Use graph neural networks to construct three undirected graphs G for the text mode, voice mode, and visual mode of each sentence t =(V t ,E t ), G a =(V a ,E a ),G v =(V v ,E v ); where G t , G a , G v They represent the text modality undirected graph, the speech modality undirected graph, and the visual modality undirected graph, respectively. V represents the set of discourse nodes, and E represents the set of adjacency relationships between the feature nodes of each sentence. S23. Use a graph neural network to construct a graph for all text, speech, and visual modalities of all utterances, and for all sample data of the same modality, connect each sample node to every other sample node. S24. Calculate the similarity A between sample nodes based on the Pearson correlation coefficient xy : Among them, x i and y i Respectively represent the values ​​of the two samples at the i-th feature, and Represents the mean of all features of the two samples respectively; S25. Add the initial residual connection and identity mapping to the GCN network to aggregate context features. Let is the graph convolution matrix of the undirected graph G: Among them, A represents the adjacency matrix, D represents the diagonal matrix of the graph G, and I represents the identity matrix; the operation between the multi-layer graph convolutional network is shown as follows: Among them, α and β (l) Represent two hyperparameters, σ represents the activation function, W (l) represents the learnable weight matrix; set up Where η represents a hyperparameter, the first layer H (0) The residual connection is added to the representation of each layer and use the unit map I plus the weight matrix W (l) , and obtain the feature representation of the current layer that aggregates the information of the previous layer.

3. A multimodal dialogue emotion recognition method according to claim 2, characterized in that: The method of linearly transforming the text modality, voice modality, and visual modality features of each sentence in the conversation and then fusing the features between the modalities is: F i m =concat[F i t ,F i a ,F i v ] in, and Respectively represent the text modality, acoustic modality and visual modality aggregated conversation context information features, F i t , F i a , F i v Respectively and After linear transformation, the feature W t 、W a and W v is the training weight parameter, b t 、b a and b v is the bias parameter, F i m It represents the fusion features of text modality, speech modality and visual modality, and concat[·] represents the concatenation operation.

4. A multimodal dialogue emotion recognition method according to claim 3, characterized in that: The classifier calculation process is shown in the formula: F i 'k =σ(W k F i k +b k ) Among them, k∈{t,a,v,m}, F i k represents a single modal feature or fusion feature entering the classifier, σ(·) represents the activation function, F i 'k Represents the features after the activation function calculation, Softmax(·) normalizes the input features into probability distribution, P i k Indicates the probability that the sample belongs to each category, W k and represents the weight parameter of training, b k and represents the bias parameter, represents the classification result, argmax(·) calculates the category with the highest classification probability, and c represents the number of categories in all samples.

5. A multimodal dialogue emotion recognition method according to claim 4, characterized in that: The classification loss function is calculated as shown in the formula: Where N is the number of dialogues, c(i) is the number of utterances in the dialogue, k∈{t,a,v,m}; α k represents the weight of each subtask loss, represents the probability distribution of emotion labels predicted by utterance j in conversation i in subtask k, y i,j represents the expected class label of utterance j in dialogue i, λ represents the L2 regularization weight, and θ represents the set of all training parameters.

6. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a multimodal conversation emotion recognition method according to any one of claims 1 to 5 when executing the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multimodal dialogue emotion recognition method described in any one of claims 1 to 5.

Images