Multi-modal sentiment recognition method for dialogue scenario

Abstract

The application provides a multi-modal sentiment recognition method for a dialogue scene, and relates to the technical field of computer processing. The method comprises the following steps: obtaining a to-be-tested sample comprising an audio file, a video file and a text file recording the same dialogue scene; performing sentiment recognition on the to-be-tested sample through a preset multi-modal sentiment recognition strategy to obtain a first recognition result comprising a single-modal prediction probability distribution corresponding to the to-be-tested sample and a second recognition result comprising a multi-modal fusion prediction probability distribution corresponding to the to-be-tested sample; and splicing the first recognition result and the second recognition result to obtain a sentiment prediction result representing the sentiment state of a current sentence. In this way, the sentence in the dialogue scene is subjected to double sentiment recognition in single mode and multi-mode, and the sentiment prediction result is obtained by splicing the two, which can better adapt to the influence of a complex dialogue scene on the sentiment recognition result and improve the problem of large sentiment recognition error and low correlation with sentiment influencing factors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, in particular to a multi-modal emotion recognition method for a dialogue scene. BACKGROUND

[0002] With the rapid development of social media and artificial intelligence, the concept of intelligence has gradually penetrated into people's hearts and exists in all aspects of life. Among them, the automatic recognition of human emotional state has been applied in many life fields, such as assisting in monitoring patients or the elderly, assisting in traffic warning, etc.

[0003] At present, the emotion state recognition for dialogue scene usually aims at the information collected by the front-end device (such as text, audio or video containing the dialogue of the detected person), extracts the features existing in the information, and recognizes the emotion state of the detected person represented by the features through a neural network. However, in actual application, the dialogue of people often presents different emotional states of the dialogue person due to factors such as context, interactive relationship of dialogue person, context relevance, etc. Therefore, the above-mentioned single-modal emotion recognition mode for text, audio or video information cannot adapt to the influence of complex actual dialogue scene on emotion recognition results, and there is a problem of large emotion recognition error and low relevance to emotion influencing factors. SUMMARY

[0004] Therefore, the purpose of the embodiments of the present application is to provide a multi-modal emotion recognition method and device for a dialogue scene, an electronic device and a storage medium, which can improve the problem that the traditional emotion recognition mode cannot adapt to the influence of complex actual dialogue scene on emotion recognition results, and there is a large emotion recognition error and low relevance to emotion influencing factors.

[0005] To achieve the above technical purpose, the technical solution adopted by the present application is as follows:

[0006] The embodiments of the present application provide a multi-modal emotion recognition method for a dialogue scene, which comprises:

[0007] Obtaining a to-be-tested sample, the to-be-tested sample comprising an audio file, a video file and a text file recording the same dialogue scene;

[0008] Performing emotion recognition on the to-be-tested sample through a preset multi-modal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively, the first recognition result comprising a single-modal prediction probability distribution corresponding to the to-be-tested sample, and the second recognition result comprising a multi-modal fusion prediction probability distribution corresponding to the to-be-tested sample;

[0009] The first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested.

[0010] In some optional embodiments, emotion recognition is performed on the sample to be tested by presetting a multimodal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively, including:

[0011] Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set;

[0012] Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set;

[0013] Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features respectively;

[0014] Performing emotion recognition on the convolution operation result to obtain the first recognition result, where the first recognition result includes the unimodal predicted probability distribution corresponding to different modalities of each sentence in the test sample;

[0015] According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are respectively spliced ​​by a preset multi-head attention splicing strategy to obtain a multimodal feature;

[0016] Emotion recognition is performed on the multimodal features to obtain the second recognition result, where the second recognition result includes the multimodal fusion prediction probability distribution corresponding to each sentence in the sample to be tested.

[0017] In some optional implementations, performing feature encoding on the feature set to obtain encoding features corresponding to the feature set includes:

[0018] Each sentence in the feature set is feature encoded by a bidirectional gated recurrent unit:

[0019]

[0020] Where, represents a bidirectional gated recurrent unit in the text modality, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, and Represent the text features, audio features and video features of the i-th sentence respectively, and Represent the encoding features corresponding to the text features, audio features, and video features of the i-th sentence, respectively. and Represents the encoding features corresponding to the text features, audio features, and video features of the i+1th or i-1th sentence respectively.

[0021] In some optional implementations, performing a graph convolution operation on the encoding feature to obtain a graph convolution operation result includes:

[0022] According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows:

[0023]

[0024] Where h′ i represents the convolution operation result corresponding to the encoding feature of the i-th sentence, h i Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and A set of softmax represents the first activation function, W1, W2, W3, W4 represent trainable parameter matrices, h j Represents the encoding features of the jth sentence in text, acoustic and visual modalities and A collection of represents the set of all sentences in the same dialogue as the i-th sentence, and d represents the dimension of the encoded features.

[0025] In some optional implementations, performing emotion recognition on the convolution operation result to obtain the first recognition result includes:

[0026] Perform emotion recognition on the convolution operation result:

[0027]

[0028] Where, Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and The corresponding convolution operation result, W l and W uc represents the trainable weight matrix, b l and b uc represents a trainable bias term, express The feature vector after being processed by the second activation function ReLU, represents the predicted probability distribution of the i-th sentence in the first recognition result in text, acoustic and visual modalities, Softmax represents the first activation function, Represents the unimodal sentiment label corresponding to the i-th sentence in text, acoustic and visual modalities.

[0029] In some optional implementations, based on the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​by a preset multi-head attention splicing strategy to obtain multimodal features, including:

[0030] Map the audio features and the video features to the text space as the attention output of a single attention head:

[0031]

[0032] in:

[0033]

[0034] Where H a ′、H t ′ and H v ′ represents the convolution operation results corresponding to audio features, text features and video features respectively, Q (v-t) , K (v-t) and V (v-t) represents the attention parameter obtained by mapping video features to text space, Q (a-t) , K (a-t) and V (a-t) represents the attention parameter obtained by mapping audio features to text space, and Represent the trainable weight matrices respectively;

[0035] The attention output corresponding to the audio feature and the attention output corresponding to the video feature are respectively spliced ​​to obtain the attention features of the audio feature and the video feature in the text space:

[0036]

[0037] Where A (a-t) Represents the attention feature of audio features in the text space, A (v-t) Represents the attention features of video features in the text space, Concat represents splicing, N att Indicates the number of attention heads;

[0038] The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are spliced ​​to obtain the multimodal feature:

[0039] H m =Concat(A (a-t) ,H′ t ,A (v-t) )

[0040] Where H m Represents multimodal features, H′ t The convolution operation result corresponding to the encoded features of the text modality.

[0041] In some optional implementations, performing emotion recognition on the multimodal features to obtain the second recognition result includes:

[0042] Each sentence in the sample to be tested is regarded as a node. Based on the multimodal features corresponding to the node, the connection relationship between the current node and other nodes in the same conversation is identified to obtain the relationship features:

[0043]

[0044] Where, represents the relationship feature of the i-th node, represents the multimodal features corresponding to the i-th node, Represents the multimodal features corresponding to the jth node, j∈n r (i), n r (i) represents the set of all nodes in the same conversation with the i-th node, |n r (i)| is the normalization constant, θ root and θ r Represents trainable parameters;

[0045] Obtaining weight parameters between the current node and other nodes in the same conversation;

[0046] Determine the global feature of each node based on the relationship feature corresponding to each node and the weight parameter:

[0047]

[0048] Where g i represents the global feature corresponding to the i-th node, represents the relationship feature corresponding to the i-th node, represents the relationship feature corresponding to the jth node, j∈n(i), n(i) represents the set of all nodes in the same conversation with the i-th node, and represents the trainable weight matrix, e i,j represents the weight parameter;

[0049] The second recognition result is determined according to the relationship feature and the global feature.

[0050] In some optional implementations, obtaining weight parameters between the current node and other nodes in the same conversation includes:

[0051] Determine the weight parameter according to the relationship feature corresponding to each node:

[0052]

[0053] Where, e i,j Represents the weight parameter between node i and node j.

[0054] In some optional implementations, determining the second recognition result according to the relationship feature and the global feature includes:

[0055] Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node:

[0056]

[0057] Where, e i Indicates the feature representation of the sentence corresponding to the current node;

[0058] Emotion recognition is performed based on the feature representation to obtain the second recognition result:

[0059] l i =RELU(W j e i +b j )

[0060]

[0061] Where W j and W k represents the trainable weight matrix, b j and b k Represents a trainable bias term, softmax represents the first activation function, RELU represents the second activation function, l i Represents the feature vector after being processed by the second activation function RELU, Indicates the second recognition result.

[0062] In some optional implementations, the first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the test sample, including:

[0063] Perform a weighted summation on the first recognition result and the second recognition result to obtain the emotion prediction result:

[0064]

[0065] Where, represents the sentiment prediction result, represents the predicted probability distribution of the i-th sentence in the text, acoustic, and visual modalities in the first recognition result, and β, γ, η, and ζ represent the preset hyperparameters.

[0066] The invention adopting the above technical solution has the following advantages:

[0067] In the technical solution provided in this application, a test sample is first obtained, which records a conversation in a text, audio, or video file; then, through a preset multimodal emotion recognition strategy, emotion recognition is performed on the test sample to obtain a first recognition result including a unimodal prediction probability distribution corresponding to the test sample, and a second recognition result including a multimodal fusion prediction probability distribution corresponding to the test sample; then, the first recognition result and the second recognition result are spliced ​​to obtain an emotion prediction result that characterizes the emotional state of the current sentence in the test sample. In this way, by performing unimodal and multimodal dual emotion recognition on the sentences in the dialogue scene and combining the two to obtain the final emotion prediction result, it can better adapt to the impact of complex actual dialogue scenes on emotion recognition results and improve the problems of large emotion recognition errors and low correlation with emotion influencing factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present application may be further illustrated by the non-limiting embodiments provided in the accompanying drawings. It should be understood that the following drawings illustrate only certain embodiments of the present application and are therefore not to be construed as limiting the scope of the present application. It is understood that a person skilled in the art can derive other relevant drawings from these drawings without inventive effort.

[0069] Figure 1 A flowchart of a multimodal emotion recognition method for conversational scenarios provided in an embodiment of the present application.

[0070] Figure 2 Schematic diagram of the feature extraction process of text files provided in the embodiment of the present application.

[0071] Figure 3 Schematic diagram of the feature extraction process of audio files provided in an embodiment of the present application.

[0072] Figure 4 Schematic diagram of the feature extraction process of video files provided in an embodiment of the present application.

[0073] Figure 5 Schematic diagram of the processing flow of the preset multi-head attention splicing strategy provided in an embodiment of the present application.

[0074] Figure 6 This is a schematic diagram of the structure of the MMGCN model provided in the embodiments of the present application. DETAILED DESCRIPTION

[0075] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts in the drawings or descriptions are numbered the same. Implementations not shown or described in the drawings are known to those of ordinary skill in the art. In the description of this application, the terms "first," "second," etc. are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance.

[0076] Please refer to Figure 1 The embodiment of the present application provides a multimodal emotion recognition method for a conversation scenario. The multimodal emotion recognition method for a conversation scenario may include the following steps:

[0077] Step 110: obtaining a sample to be tested, wherein the sample to be tested includes an audio file, a video file, and a text file recording the same conversation scene;

[0078] Step 120: Perform emotion recognition on the sample to be tested using a preset multimodal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively. The first recognition result includes a single-modal prediction probability distribution corresponding to the sample to be tested, and the second recognition result includes a multimodal fusion prediction probability distribution corresponding to the sample to be tested.

[0079] Step 130 : Concatenate the first recognition result and the second recognition result to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested.

[0080] In the above-mentioned embodiment, first, a test sample is obtained in which a conversation is recorded in a text, audio, or video file; then, through a preset multimodal emotion recognition strategy, emotion recognition is performed on the test sample to obtain a first recognition result including a unimodal prediction probability distribution corresponding to the test sample, and a second recognition result including a multimodal fusion prediction probability distribution corresponding to the test sample; then, the first recognition result and the second recognition result are spliced ​​to obtain an emotion prediction result that characterizes the emotional state of the current sentence in the test sample. In this way, by performing unimodal and multimodal dual emotion recognition on the sentences in the dialogue scene and combining the two to obtain the final emotion prediction result, it is possible to better adapt to the impact of complex actual dialogue scenes on the emotion recognition results and improve the problems of large emotion recognition errors and low correlation with emotion influencing factors.

[0081] The following is a detailed description of the steps of the multimodal emotion recognition method for dialogue scenarios:

[0082] In step 110, the sample to be tested may be speech (or conversation, where a conversation contains multiple sentences) of an interlocutor in a conversation scenario captured by a pre-set front-end device (e.g., a microphone, a camera, etc.). The captured information may be a video file, an audio file, etc. The text file may be captured by capturing the interlocutor's audio file through a microphone and converting the audio file into a text format. The conversion method from audio file to text file may be conventional in the field of artificial intelligence technology and will not be elaborated here.

[0083] In step 120, emotion recognition is performed on the sample to be tested by a preset multimodal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively, which may include:

[0084] Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set;

[0085] Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set;

[0086] Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features respectively;

[0087] Performing emotion recognition on the convolution operation result to obtain the first recognition result, where the first recognition result includes the unimodal predicted probability distribution corresponding to different modalities of each sentence in the test sample;

[0088] According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features and the text features are spliced respectively by a preset multi-head attention splicing strategy, to obtain multi-modal features.

[0089] The multi-modal features are subjected to sentiment recognition to obtain the second recognition result, which includes the multi-modal fusion prediction probability distribution corresponding to each sentence in the to-be-tested sample.

[0090] In the embodiment, the feature extraction on the to-be-tested sample can include:

[0091] For the text file, a pre-trained Roberta model is used to extract the text features.

[0092] For the audio file, an OpenSmile toolkit with IS10 configuration is used to extract the audio features.

[0093] For the video file, a pre-trained DenseNet model is used to extract the video features.

[0094] Specifically, for the text file, please refer to Figure 2 , first, the data is preprocessed, and the original text file is subjected to BPE (Byte Pair Encoder) encoding processing to obtain a more fine-grained word representation. Then, the Fairseq library is used for preprocessing, and the BPE encoded text file is converted into a binary format required by the model training. Specifically, this process includes downloading the encoder, word table and dictionary file of the GPT-2 model, and using multi-threading to BPE encode the original text files of the training set, validation set and test set, and finally generating the binary format training data preprocessed by Fairseq. In the training stage, the RoBERTa-large model is used to train the preprocessed text file (i.e. training data) using the pre-labeled sentiment labels. The goal of this stage is to fine-tune the parameters of the RoBERTa-large model by repeatedly iterating on the training data set to adapt to the sentiment recognition task. RoBERTa-large contains 24 hidden layers, each layer has 16 attention heads, a total of 355M parameters, and finally outputs a 1024-dimensional tensor as the text feature.

[0095] For the audio file, please refer to Figure 3First, start openSMILE, set the IS10 configuration file, and read the speech modality data (i.e., audio file) of each utterance according to the file name. Then, preprocess the audio file, including pre-emphasis, framing, windowing, and other steps. In the audio file, the original speech signal is usually a continuous waveform representing the speaker's voice. Then, various features in the audio data are extracted through openSMILE using the feature set in the IS10 configuration file. The set contains 1582 features, which are derived from 34 basic low-level descriptors (LLDs) and their corresponding 34 incremental coefficients. Each LLD contour is applied with 21 functions (1428 features). In addition, 19 functions (152 features) are applied to the four pitch-based LLDs and their four incremental coefficient contours, and the number of pitch onsets (pseudosyllables) and the total duration of the input (2 features) are also added. LLD includes but is not limited to acoustic features such as sound loudness, MFCC, voiced characteristics, F0 fundamental frequency, etc. Finally, by batch processing the entire dataset, the audio features of all utterances are obtained, and each utterance is represented as a 1582-dimensional feature vector.

[0096] For video files, see Figure 4 , a DenseNet model is used to extract visual features from video files (video files recording conversation scenes can be composed of multiple image frames). DenseNet (Densely Connected Convolutional Networks) is a CNN architecture. In DenseNet, each layer receives additional input from all previous layers and passes its own feature map to all subsequent layers. The grayscale images are scaled to a uniform size (48×48×1) to ensure that they can be effectively processed by the DenseNet model. The scaled and normalized images are then input into the pre-trained DenseNet model. DenseNet extracts high-level features from the image through its deep convolutional network structure, which can represent the emotional content in the image. The pre-trained model has learned a large number of visual feature representations, so it can effectively identify emotion-related features from the input image. Ultimately, the DenseNet model outputs a 1×342-dimensional image emotion vector feature as a video feature.

[0097] In this embodiment, feature encoding is performed on the feature set to obtain encoding features corresponding to the feature set, which may include:

[0098] Each sentence in the feature set is feature encoded by a bidirectional gated recurrent unit:

[0099]

[0100] Where, represents a bidirectional gated recurrent unit in the text modality, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, and Represent the text features, audio features and video features of the i-th sentence respectively, and Represent the encoding features corresponding to the text features, audio features, and video features of the i-th sentence, respectively. and Represents the encoding features corresponding to the text features, audio features, and video features of the i+1th or i-1th sentence respectively.

[0101] In this embodiment, performing a graph convolution operation on the coding feature to obtain a graph convolution operation result may include:

[0102] According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows:

[0103]

[0104]

[0105] Where h′ i represents the convolution operation result corresponding to the encoding feature of the i-th sentence, h i Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and A set of softmax represents the first activation function, W1, W2, W3, W4 represent trainable parameter matrices, h j Represents the encoding features of the jth sentence in text, acoustic and visual modalities and A collection of represents the set of all sentences in the same dialogue as the i-th sentence, and d represents the dimension of the encoded features.

[0106] In this embodiment, please refer to Figure 6 In the local graph representation part, in order to capture the feature relationship of a sentence in each modality, a local graph is constructed for each modality. For a dialogue scene containing N sentences, it is represented by a directed graph as in, and ε represent nodes (a sentence is considered a node) and adjacency matrices, respectively, and R represents the type of relationship connecting nodes. Each node in the graph is initialized using the encoding features corresponding to the node in the text, acoustic, and visual modalities.

[0107] For the connection relationship between nodes in the local graph, if there are M (M≥2) interlocutors in the dialogue scene, the dialogue relationship between the interlocutors can be divided into the speaker himself (S1) and other people in the dialogue scene (S2) (refer to Figure 6 In this embodiment, two interlocutors are used as an example. Based on this, four types of undirected dynamic information transmission relationships are defined:

[0108] 1. The speaker's own utterance self-connection (s1→s1);

[0109] 2. Self-connections between utterances of other speakers (s2→s2);

[0110] 3. The connection relationship between the speaker itself and other speakers (s1→s2);

[0111] 4. The connection relationship between other speakers and the speaker itself (s2→s1).

[0112] Furthermore, if we focus on the temporal relationships within a conversation, eight types of relationships are formed, including the relationship between the current statement and the previous statement (forward connections) and the relationship between the current statement and future statements (backward connections). Furthermore, the backward connection (s1 → s2) is not symmetrical with the forward connection (s2 → s1) because they have different start and end nodes.

[0113] Based on the above-mentioned local graph structure, this embodiment performs graph convolution on the coding features using the above-mentioned graph convolution operation formula to obtain a convolution operation result, thereby converting the coding features that are not related to the interlocutor into feature vectors related to the interlocutor, and obtaining intra-modal features (i.e., the convolution operation result) corresponding to the audio features, video features, and text features containing the interlocutor information.

[0114] In this embodiment, performing emotion recognition on the convolution operation result to obtain the first recognition result may include:

[0115] Perform emotion recognition on the convolution operation result:

[0116]

[0117] Where, Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and The corresponding convolution operation result, W l and W ucrepresents the trainable weight matrix, b l and b uc represents a trainable bias term, express The feature vector after being processed by the second activation function ReLU, represents the predicted probability distribution of the i-th sentence in the first recognition result in text, acoustic and visual modalities, Softmax represents the first activation function, Represents the unimodal sentiment label corresponding to the i-th sentence in text, acoustic and visual modalities.

[0118] In this embodiment, based on the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​by a preset multi-head attention splicing strategy to obtain multimodal features, which may include:

[0119] Map the audio features and the video features to the text space as the attention output of a single attention head:

[0120]

[0121] in:

[0122]

[0123] Where H a ′、H t ′ and H v ′ represents the convolution operation results corresponding to audio features, text features and video features respectively, Q (v-t) , K (v-t) and V (v-t) represents the attention parameter obtained by mapping video features to text space, Q (a-t) , K (a-t) and V (a-t) represents the attention parameter obtained by mapping audio features to text space, and Represent the trainable weight matrices respectively;

[0124] The attention output corresponding to the audio feature and the attention output corresponding to the video feature are respectively spliced ​​to obtain the attention features of the audio feature and the video feature in the text space:

[0125]

[0126] Where A (a-t) Represents the attention feature of audio features in the text space, A (v-t)Represents the attention features of video features in the text space, Concat represents splicing, N att Indicates the number of attention heads;

[0127] The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are spliced ​​to obtain the multimodal feature:

[0128] H m =Concat(A (a-t) ,H′ t ,A (v-t) ) (8)

[0129] Where H m Represents multimodal features, H′ t The convolution operation result corresponding to the encoded features of the text modality.

[0130] In this embodiment, referring to Figure 5 The processing flow of the preset multi-head attention splicing strategy is to use two attention modules to perform convolution operation on the input result H a ′、H t ′ and H v ′ Perform feature conversion through equations (5) and (6) to map the audio features and video features to the text space. Then, the attention output corresponding to the audio features and the attention output corresponding to the video features are spliced ​​separately through the multi-head attention mechanism through equation (7) to obtain the attention features of the audio features in the text space and the attention features of the video features in the text space. Finally, the convolution operation results corresponding to the attention features of the audio features in the text space, the attention features of the video features in the text space, and the encoding features of the text modality are spliced ​​to obtain the multimodal feature H. m .

[0131] In this embodiment, performing emotion recognition on the multimodal features to obtain the second recognition result may include:

[0132] Each sentence in the sample to be tested is regarded as a node. Based on the multimodal features corresponding to the node, the connection relationship between the current node and other nodes in the same conversation is identified to obtain the relationship features:

[0133]

[0134] Where, represents the relationship feature of the i-th node, represents the multimodal features corresponding to the i-th node, Represents the multimodal features corresponding to the jth node, j∈nr (i), n r (i) represents a set of all nodes within the same segment of dialogue with the i-th node, |n r (i) | is a normalization constant, θ root and θ r represent trainable parameters;

[0135] obtaining a weight parameter between the current node and other nodes in the same segment of dialogue;

[0136] determining a global feature of each node according to the relationship feature corresponding to each node and the weight parameter;

[0137]

[0138] wherein g i represents the global feature corresponding to the i-th node, represents the relationship feature corresponding to the i-th node, represents the relationship feature corresponding to the j-th node, j ∈ n(i), n(i) represents a set of all nodes within the same segment of dialogue with the i-th node, and represents a trainable weight matrix, e i,j represents a weight parameter;

[0139] determining the second recognition result according to the relationship feature and the global feature.

[0140] In the embodiment, please refer to the global graph representation part in Figure 6 The structure relationship of the global graph is the same as that of the local graph, and there are four kinds of undirected connection relationships between the dialogues, and each connection relationship has two kinds of time connection types, a total of eight connection relationships. In the embodiment, the relationship fusion information of the current node and other nodes within the same segment of dialogue is captured through the above formula (9), and the relationship fusion information is refined and enhanced through the above formula (10), so as to realize multi-modal feature fusion of different information.

[0141] Wherein, obtaining the weight parameter between the current node and other nodes in the same segment of dialogue can include:

[0142] determining the weight parameter according to the relationship feature corresponding to each node;

[0143]

[0144] wherein e i,j represents the weight parameter between node i and node j.

[0145] Determining the second recognition result according to the relationship feature and the global feature may include:

[0146] Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node:

[0147]

[0148] Where, e i Indicates the feature representation of the sentence corresponding to the current node;

[0149] Emotion recognition is performed based on the feature representation to obtain the second recognition result:

[0150] l i =RELU(W j w i +b j ) (13)

[0151]

[0152] Where W j and W k represents the trainable weight matrix, b j and b k Represents a trainable bias term, softmax represents the first activation function, RELU represents the second activation function, l i Represents the feature vector after being processed by the second activation function RELU, Indicates the second recognition result.

[0153] In step 130, the first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the test sample, which may include:

[0154] Perform a weighted summation on the first recognition result and the second recognition result to obtain the emotion prediction result:

[0155]

[0156] Where, represents the sentiment prediction result, represents the predicted probability distribution of the i-th sentence in the text, acoustic, and visual modalities in the first recognition result, and β, γ, η, and ζ represent the preset hyperparameters.

[0157] In summary, please refer to Figure 6 In practical applications, we can Figure 6The MMGCN (multimodal emotion recognition) model shown in the figure implements the multimodal emotion recognition method for dialogue scenarios proposed in this application. Among them, the MMGCN model includes a unimodal context editing module, a local graph module, a unimodal classification and fusion module, a global graph module and a multimodal classifier. Combined with the aforementioned multimodal emotion recognition method for dialogue scenarios, the audio files, text files and video files recording the dialogue scenes are input into the MMGCN model, and then the unimodal context editing module is used to extract features from each input file to obtain the encoding features under each modality. and Then, the local graph module is used to perform graph convolution on the encoded features to obtain the convolution result h′ i , that is, the convolution operation results corresponding to each mode and Then, the unimodal classifier in the unimodal classification and fusion module is used to perform sentiment prediction on the convolution operation results under the three modes, and the unimodal prediction probability distribution corresponding to each sentence in the dialogue scene is obtained. That is, the first recognition result and the unimodal emotion label representing the unimodal emotion prediction result Then, through the cross-modal fusion attention module in the single-modal classification and fusion module, based on the preset multi-head attention splicing strategy, the intra-modal features corresponding to the audio features, video features and text features are spliced ​​to obtain the multimodal feature H m , and then extract the relationship features of multimodal features through the global graph module and global feature g i , and connect the relational features and global features through the multimodal classifier to obtain the feature representation e corresponding to each sentence in the dialogue scene i , and according to e i Determine the second recognition result Finally, the first recognition result output by the unimodal classifier And the second recognition result output by the multimodal classifier Perform weighted summation to obtain the label representing the sentiment prediction result In this way, information from the three modalities of text, audio, and video can be effectively captured, and the emotional content contained in the conversation between the interlocutors can be more comprehensively represented. Based on the preset multi-head attention splicing strategy, the limitations of feature alignment and fusion between different modalities can be improved, thereby improving the accuracy of emotion recognition for dialogue scenarios.

[0158] To demonstrate the technical advantages of the embodiments of this application, the MMGCN model is evaluated using two benchmark datasets: IEMOCAP and MELD. Both datasets are multimodal datasets that contain textual, visual, and acoustic information for each sentence in each conversation.

[0159] The IEMOCAP dataset is an interactive emotional binary motion capture database released by the SAIL laboratory at the University of Southern California. It contains two-way conversation videos between ten different interlocutors (five males and five females), including 12 hours of audiovisual data with text transcription. The dataset consists of five chapters, each containing a conversation video between two interlocutors. Each video contains a binary dialogue, and each dialogue is segmented into individual sentences. The sentence emotion labels include six categories: happy, sad, neutral, excited, frustrated, and angry.

[0160] MELD is an extension of the Emotion Lines dataset, a purely textual conversation dataset, designed specifically for emotion recognition in conversations. MELD expands on Emotion Lines by adding speech and visual modalities and provides three higher-quality modality-aligned conversation data. Unlike binary conversation datasets such as IEMOCAP, MELD has more than two interlocutors in a conversation. The dataset contains a total of 13,708 sentences, 1,433 conversations, and a total of 304 different interlocutors. Each sentence is annotated with an emotion label, emotion polarity, and interlocutor information. Each sentence in the conversation is annotated with one of seven emotion labels, including anger, disgust, fear, happiness, neutrality, sadness, and surprise.

[0161] This application presents a comparative analysis of different modalities, demonstrating the performance of the proposed MMGCN model across two benchmark datasets. As shown in Table 1, the best single-modal performance is achieved in the text modality, while the worst is achieved in the visual modality. Adding acoustic and visual modalities can provide additional performance improvements over the text modality alone.

[0162] Table 1 Performance of MMGCN under different multimodal settings

[0163]

[0164] Table 2 below shows a comparative analysis of the proposed MMGCN model and other existing emotion recognition models. The MMGCN model achieved state-of-the-art results on two multimodal emotion recognition datasets. On the IEMOCAP dataset, the MMGCN model achieved 70.12% accuracy and 70.22% F1 score. On MELD, the MMGCN model also outperformed all other models, improving F1 by 5.72% over MM-DFN, achieving 66.21% accuracy and 65.18% weighted F1 score.

[0165] Table 2 Model performance comparison

[0166]

[0167]

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0169] In the embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The method embodiments described above are merely schematic. For example, the flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0170] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal emotion recognition method for dialogue scenarios, characterized by: The method comprises: Obtaining a sample to be tested, wherein the sample to be tested includes an audio file, a video file, and a text file recording the same conversation scene; By presetting a multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result and a second recognition result are obtained respectively, wherein the first recognition result includes a single-modal prediction probability distribution corresponding to the sample to be tested, and the second recognition result includes a multimodal fusion prediction probability distribution corresponding to the sample to be tested; splicing the first recognition result and the second recognition result to obtain an emotion prediction result representing the emotion state of the current sentence in the test sample; The method includes performing emotion recognition on the sample to be tested by presetting a multimodal emotion recognition strategy, and obtaining a first recognition result and a second recognition result, respectively. Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set; Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set; Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features respectively; Performing emotion recognition on the convolution operation result to obtain the first recognition result, where the first recognition result includes the unimodal predicted probability distribution corresponding to different modalities of each sentence in the test sample; According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are respectively spliced ​​by a preset multi-head attention splicing strategy to obtain a multimodal feature; Performing emotion recognition on the multimodal features to obtain a second recognition result, where the second recognition result includes the multimodal fusion prediction probability distribution corresponding to each sentence in the test sample; According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​by a preset multi-head attention splicing strategy to obtain multimodal features, including: Map the audio features and the video features to the text space as the attention output of a single attention head: ； in: ； Where, 、 and Represents the convolution operation results corresponding to audio features, text features, and video features respectively, 、 and represents the attention parameter obtained by mapping video features to text space, 、 and represents the attention parameter obtained by mapping audio features to text space, 、 、 、 、 and Represent the trainable weight matrices respectively; The attention output corresponding to the audio feature and the attention output corresponding to the video feature are respectively spliced ​​to obtain the attention features of the audio feature and the video feature in the text space: ； Where, Represents the attention features of audio features in the text space, Represents the attention features of video features in the text space, Indicates splicing, Indicates the number of attention heads; The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are spliced ​​to obtain the multimodal feature: ； Where, Represents multimodal features, The convolution operation result corresponding to the encoded features of the text modality; The step of performing emotion recognition on the multimodal features to obtain the second recognition result includes: Each sentence in the sample to be tested is regarded as a node. Based on the multimodal features corresponding to the node, the connection relationship between the current node and other nodes in the same conversation is identified to obtain the relationship features: ； Where, Indicates the The relationship characteristics of the nodes, Indicates the The multimodal features corresponding to the nodes, Indicates the The multimodal features corresponding to the nodes, , Indicates The set of all nodes in the same conversation. is the normalization constant, and Represents trainable parameters; Obtaining weight parameters between the current node and other nodes in the same conversation; Determine the global feature of each node based on the relationship feature corresponding to each node and the weight parameter: ； Where, Indicates the The global features corresponding to the nodes, Indicates the The relationship features corresponding to the nodes, Indicates the The relationship features corresponding to the nodes, , Indicates The set of all nodes in the same conversation. and represents the trainable weight matrix, represents the weight parameter; The second recognition result is determined according to the relationship feature and the global feature.

2. The method according to claim 1, characterized in that Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set includes: Each sentence in the feature set is feature encoded by a bidirectional gated recurrent unit: ； Where, represents a bidirectional gated recurrent unit in the text modality, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, 、 and Respectively represent The text features, audio features and video features of each sentence, 、 and Respectively represent The encoding features corresponding to the text features, audio features and video features of each sentence, 、 and Respectively represent or The encoding features corresponding to the text features, audio features and video features of a sentence.

3. The method according to claim 1, characterized in that Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, including: According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows: ； ； Where, Indicates the The convolution operation result corresponding to the encoding feature of each sentence, Indicates the Encoding features of a sentence in text, acoustic, and visual modalities 、 and A collection of represents the first activation function, 、 、 、 represents the trainable parameter matrix, represents the context encoding vector, Indicates the Encoding features of a sentence in text, acoustic, and visual modalities 、 and A collection of , Indicates the The set of all sentences in the same conversation. Indicates the dimension of the encoded features.

4. The method according to claim 1, wherein Performing emotion recognition on the convolution operation result to obtain the first recognition result includes: Perform emotion recognition on the convolution operation result: ； Where, Indicates the Encoding features of sentences in text, acoustic, and visual modalities 、 and The corresponding convolution operation result is, and represents the trainable weight matrix, and represents the trainable bias term, express After the second activation function The processed feature vector, Indicates the first recognition result The predicted probability distribution of a sentence in text, acoustic and visual modalities, represents the first activation function, Indicates the Unimodal sentiment labels corresponding to sentences in textual, acoustic, and visual modalities.

5. The method according to claim 1, wherein Obtaining weight parameters between the current node and other nodes in the same conversation, including: Determine the weight parameter according to the relationship feature corresponding to each node: ； Where, Representation node With node The weight parameter between .

6. The method according to claim 1, characterized in that Determining the second recognition result according to the relationship feature and the global feature includes: Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node: ； Where, Indicates the feature representation of the sentence corresponding to the current node; Emotion recognition is performed based on the feature representation to obtain the second recognition result: ； ； Where, and represents the trainable weight matrix, and represents the trainable bias term, represents the first activation function, represents the second activation function, Indicates that after the second activation function The processed feature vector, Indicates the second recognition result.

7. The method according to claim 1, characterized in that The first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested, including: Perform a weighted summation on the first recognition result and the second recognition result to obtain the emotion prediction result: ； Where, represents the sentiment prediction result, 、 、 Indicates that in the first recognition result, The predicted probability distribution of a sentence in text, acoustic and visual modalities, 、 、 、 Represents the preset hyperparameters.

Images