Dialogue scene-oriented multi-modal emotion recognition method

Abstract

The invention provides a dialogue scene-oriented multi-modal emotion recognition method, and relates to the technical field of computer processing. The method comprises the following steps: acquiring a to-be-tested sample including an audio file, a video file and a text file which record the same dialogue scene; performing emotion recognition on the to-be-tested sample through a preset multi-modal emotion recognition strategy, and respectively obtaining a first recognition result comprising single-modal prediction probability distribution corresponding to the to-be-tested sample and a second recognition result comprising multi-modal fusion prediction probability distribution corresponding to the to-be-tested sample; and splicing the first recognition result and the second recognition result to obtain an emotion prediction result representing the emotion state of the current statement. Therefore, the sentences in the dialogue scene are subjected to single-mode and multi-mode dual emotion recognition, and the sentences are spliced to obtain the emotion prediction result, so that the influence of the complex dialogue scene on the emotion recognition result can be better adapted, and the problems that the emotion recognition error is relatively large and the relevance with emotion influence factors is relatively low are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a multimodal emotion recognition method for dialogue scenarios. Background Art

[0002] With the rapid development of social media and artificial intelligence, the concept of intelligence has gradually become popular and exists in all aspects of life. Among them, the automatic recognition of human emotional state has been used in many areas of life, such as assisting the care of patients or the elderly, assisting traffic warnings, etc.

[0003] At present, emotional state recognition for dialogue scenarios usually targets the information collected by the front-end device (such as text, audio or video containing the dialogue of the person being detected), extracts the features in the information, and identifies the emotional state of the person being detected represented by the features through a neural network. However, in actual applications, human conversations often present different emotional states of the interlocutors due to factors such as context, the interactive relationship between the interlocutors, and contextual relevance. Therefore, the above-mentioned single-modal emotion recognition method for text, audio or video information cannot adapt to the impact of complex actual dialogue scenarios on emotion recognition results, and there are problems such as large emotion recognition errors and low correlation with emotion influencing factors. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present application is to provide a multimodal emotion recognition method, device, electronic device and storage medium for dialogue scenarios, which can improve the impact of traditional emotion recognition methods on emotion recognition results caused by complex actual dialogue scenarios, and the problems of large emotion recognition errors and low correlation with emotion influencing factors.

[0005] In order to achieve the above technical objectives, the technical solutions adopted in this application are as follows:

[0006] The present application embodiment provides a multimodal emotion recognition method for a dialogue scenario, the method comprising:

[0007] Acquire a sample to be tested, wherein the sample to be tested includes an audio file, a video file, and a text file recording the same conversation scene;

[0008] By presetting a multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result and a second recognition result are obtained respectively, wherein the first recognition result includes a single-modal prediction probability distribution corresponding to the sample to be tested, and the second recognition result includes a multimodal fusion prediction probability distribution corresponding to the sample to be tested;

[0009] The first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested.

[0010] In some optional implementations, emotion recognition is performed on the sample to be tested by presetting a multimodal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively, including:

[0011] Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set;

[0012] Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set;

[0013] Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features, respectively;

[0014] Performing emotion recognition on the convolution operation result to obtain the first recognition result, wherein the first recognition result includes the single-modal prediction probability distribution corresponding to different modalities of each sentence in the sample to be tested;

[0015] According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​respectively by a preset multi-head attention splicing strategy to obtain a multi-modal feature;

[0016] Emotion recognition is performed on the multimodal features to obtain the second recognition result, where the second recognition result includes the multimodal fusion prediction probability distribution corresponding to each sentence in the sample to be tested.

[0017] In some optional implementations, feature encoding is performed on the feature set to obtain the encoding features corresponding to the feature set, including:

[0018] Each sentence in the feature set is feature encoded through a bidirectional gated recurrent unit:

[0019]

[0020] In the formula, represents a bidirectional gated recurrent unit in text mode, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, and Represent the text features, audio features and video features of the i-th sentence respectively, and Respectively represent the encoding features corresponding to the text features, audio features, and video features of the i-th sentence, and Respectively represent the coding features corresponding to the text features, audio features, and video features of the i+1th or i-1th sentence.

[0021] In some optional implementations, performing a graph convolution operation on the encoding feature to obtain a graph convolution operation result includes:

[0022] According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows:

[0023]

[0024] In the formula, h′ i represents the convolution operation result corresponding to the encoding feature of the i-th sentence, h i Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and A collection of softmax represents the first activation function, W1, W2, W3, and W4 represent trainable parameter matrices, and h j Represents the encoding features of the jth sentence in text, acoustic and visual modalities and A collection of represents the set of all sentences in the same dialogue as the i-th sentence, and d represents the dimension of the encoded features.

[0025] In some optional implementations, performing emotion recognition on the convolution operation result to obtain the first recognition result includes:

[0026] Perform emotion recognition on the convolution operation result:

[0027]

[0028] In the formula, Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and The corresponding convolution operation result, W l and W uc represents the trainable weight matrix, b l and b uc represents a trainable bias term, express The feature vector after being processed by the second activation function ReLU, represents the predicted probability distribution of the i-th sentence in the first recognition result in text, acoustic and visual modalities, Softmax represents the first activation function, Represents the unimodal sentiment label corresponding to the i-th sentence in text, acoustic, and visual modalities.

[0029] In some optional implementations, according to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​by presetting a multi-head attention splicing strategy to obtain a multimodal feature, including:

[0030] Map the audio features and the video features to the text space as the attention output of a single attention head:

[0031]

[0032] in:

[0033]

[0034] In the formula, H a ′、H t ′ and H v ′ represents the convolution operation results corresponding to audio features, text features and video features respectively, Q (v-t) , K (v-t) and V (v-t) represents the attention parameter obtained by mapping video features to text space, Q (a-t) , K (a-t) and V (a-t) represents the attention parameter obtained by mapping audio features to text space, and Respectively represent the trainable weight matrices;

[0035] The attention output corresponding to the audio feature and the attention output corresponding to the video feature are concatenated respectively to obtain the attention features of the audio feature and the video feature in the text space:

[0036]

[0037] In the formula, A (a-t) Represents the attention feature of the audio feature in the text space, A (v-t) represents the attention feature of the video feature in the text space, Concat represents concatenation, and N att Indicates the number of attention heads;

[0038] The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are concatenated to obtain the multimodal feature:

[0039] H m =Concat(A (a-t) ,H′ t ,A (v-t) )

[0040] In the formula, H m Represents multimodal features, H′ t The convolution operation results corresponding to the encoded features representing the text modality.

[0041] In some optional implementations, performing emotion recognition on the multimodal feature to obtain the second recognition result includes:

[0042] Each sentence in the sample to be tested is taken as a node, and the connection relationship between the current node and other nodes in the same conversation is identified according to the multimodal feature corresponding to the node, so as to obtain the relationship feature:

[0043]

[0044] In the formula, represents the relationship feature of the i-th node, represents the multimodal features corresponding to the i-th node, Represents the multimodal features corresponding to the jth node, j∈n r (i), n r (i) represents the set of all nodes in the same conversation with the i-th node, |n r (i)| is the normalization constant, θ root and θ r Represents trainable parameters;

[0045] Obtaining weight parameters between the current node and other nodes in the same conversation;

[0046] According to the relationship features corresponding to each node and the weight parameter, the global features of each node are determined:

[0047]

[0048] In the formula, g i represents the global feature corresponding to the i-th node, represents the relationship feature corresponding to the i-th node, represents the relationship feature corresponding to the jth node, j∈n(i), n(i) represents the set of all nodes in the same conversation with the i-th node, and represents the trainable weight matrix, e i,j represents the weight parameter;

[0049] The second recognition result is determined according to the relationship feature and the global feature.

[0050] In some optional implementations, obtaining weight parameters between the current node and other nodes in the same conversation includes:

[0051] Determine the weight parameter according to the relationship feature corresponding to each node:

[0052]

[0053] In the formula, e i,j Represents the weight parameter between node i and node j.

[0054] In some optional implementations, determining the second recognition result according to the relationship feature and the global feature includes:

[0055] Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node:

[0056]

[0057] In the formula, e i Indicates the feature representation of the sentence corresponding to the current node;

[0058] Emotion recognition is performed according to the feature representation to obtain the second recognition result:

[0059] l i =RELU(W j e i +b j )

[0060]

[0061] Where W j and W k represents the trainable weight matrix, b j and b k represents a trainable bias term, softmax represents the first activation function, RELU represents the second activation function, l i Represents the feature vector after being processed by the second activation function RELU, Indicates the second recognition result.

[0062] In some optional implementations, the first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested, including:

[0063] The first recognition result and the second recognition result are weightedly summed to obtain the emotion prediction result:

[0064]

[0065] In the formula, represents the sentiment prediction result, represents the predicted probability distribution of the i-th sentence in the text, acoustic, and visual modalities in the first recognition result, and β, γ, η, ζ represent the preset hyperparameters.

[0066] The invention adopting the above technical solution has the following advantages:

[0067] In the technical solution provided in the present application, firstly, a sample to be tested that records a conversation in a text, audio, or video file is obtained; then, through a preset multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result including a unimodal prediction probability distribution corresponding to the sample to be tested and a second recognition result including a multimodal fusion prediction probability distribution corresponding to the sample to be tested are obtained respectively; then, the first recognition result and the second recognition result are spliced ​​to obtain an emotion prediction result that characterizes the emotion state of the current sentence in the sample to be tested. In this way, by performing unimodal and multimodal dual emotion recognition on the sentences in the dialogue scene, and combining the two to obtain the final emotion prediction result, it is possible to better adapt to the impact of complex actual dialogue scenes on emotion recognition results, and improve the problem of large emotion recognition errors and low correlation with emotion influencing factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present application may be further described by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings may be obtained based on these drawings without creative effort.

[0069] Figure 1 A flowchart of a multimodal emotion recognition method for conversation scenarios provided in an embodiment of the present application.

[0070] Figure 2 A schematic diagram of the feature extraction process of a text file provided in an embodiment of the present application.

[0071] Figure 3 A schematic diagram of the feature extraction process of an audio file provided in an embodiment of the present application.

[0072] Figure 4 A schematic diagram of the feature extraction process of a video file provided in an embodiment of the present application.

[0073] Figure 5 A schematic diagram of the processing flow of the preset multi-head attention splicing strategy provided in an embodiment of the present application.

[0074] Figure 6 A schematic diagram of the structure of the MMGCN model provided in the embodiments of the present application. DETAILED DESCRIPTION

[0075] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that in the drawings or descriptions, similar or identical parts use the same figure numbers, and the implementation methods not shown or described in the drawings are forms known to ordinary technicians in the relevant technical field. In the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0076] Please refer to Figure 1 , the embodiment of the present application provides a multimodal emotion recognition method for a dialogue scene. The multimodal emotion recognition method for a dialogue scene may include the following steps:

[0077] Step 110, obtaining a sample to be tested, wherein the sample to be tested includes an audio file, a video file, and a text file recording the same conversation scene;

[0078] Step 120, performing emotion recognition on the sample to be tested by a preset multimodal emotion recognition strategy, and obtaining a first recognition result and a second recognition result, respectively, wherein the first recognition result includes a single-modal prediction probability distribution corresponding to the sample to be tested, and the second recognition result includes a multimodal fusion prediction probability distribution corresponding to the sample to be tested;

[0079] Step 130, concatenating the first recognition result and the second recognition result to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested.

[0080] In the above implementation, firstly, a sample to be tested that records a conversation in a text, audio, or video file is obtained; then, through a preset multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result including a unimodal prediction probability distribution corresponding to the sample to be tested and a second recognition result including a multimodal fusion prediction probability distribution corresponding to the sample to be tested are obtained respectively; then, the first recognition result and the second recognition result are spliced ​​to obtain an emotion prediction result that characterizes the emotion state of the current sentence in the sample to be tested. In this way, by performing unimodal and multimodal dual emotion recognition on the sentences in the dialogue scene, and combining the two to obtain the final emotion prediction result, it is possible to better adapt to the impact of complex actual dialogue scenes on emotion recognition results, and improve the problem of large emotion recognition errors and low correlation with emotion influencing factors.

[0081] The following is a detailed description of the steps of the multimodal emotion recognition method for dialogue scenarios:

[0082] In step 110, the sample to be tested may be a speech (or dialogue, a dialogue contains multiple sentences) of the interlocutor in the dialogue scene collected by a preset front-end device (such as a microphone, a camera, etc.), and the collected information may be a video file, an audio file, etc. Among them, the collection of the text file may be to collect the interlocutor's audio file through a microphone and convert the audio file into a text format. Among them, the conversion method of the audio file to the text file may be a conventional technical means in the field of artificial intelligence technology, which will not be elaborated here.

[0083] In step 120, emotion recognition is performed on the sample to be tested by presetting a multimodal emotion recognition strategy to obtain a first recognition result and a second recognition result, respectively, which may include:

[0084] Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set;

[0085] Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set;

[0086] Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features, respectively;

[0087] Performing emotion recognition on the convolution operation result to obtain the first recognition result, wherein the first recognition result includes the single-modal prediction probability distribution corresponding to different modalities of each sentence in the sample to be tested;

[0088] According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​respectively by a preset multi-head attention splicing strategy to obtain a multi-modal feature;

[0089] Emotion recognition is performed on the multimodal features to obtain the second recognition result, where the second recognition result includes the multimodal fusion prediction probability distribution corresponding to each sentence in the sample to be tested.

[0090] In this embodiment, extracting features from the sample to be tested may include:

[0091] For the text file, extract the text features using a pre-trained Roberta model;

[0092] For the audio file, extracting the audio features using the OpenSmile toolkit with IS10 configuration;

[0093] For the video file, a pre-trained DenseNet model is used to extract the video features.

[0094] For text files, please refer to Figure 2 First, the data is preprocessed and the original text file is encoded by BPE (Byte Pair Encoder) to obtain a more fine-grained vocabulary representation. Subsequently, the Fairseq library is used for preprocessing to convert the BPE-encoded text file into the binary format required for model training. Specifically, the process includes downloading the encoder, vocabulary, and dictionary files of the GPT-2 model, and using multi-threading to perform BPE encoding on the original text files of the training set, validation set, and test set, and finally generating training data in binary format preprocessed by Fairseq. In the training phase, the RoBERTa-large model is used to train on the preprocessed text files (i.e., training data) using pre-annotated emotion labels. The goal of this phase is to fine-tune the parameters of the RoBERTalarge model to adapt to the emotion recognition task by repeatedly iterating on the training data set. RoBERTa-large contains 24 hidden layers, each with 16 attention heads, a total of 355M parameters, and finally outputs a 1024-dimensional tensor as a text feature.

[0095] For audio files, see Figure 3First, start openSMILE, set the IS10 configuration file, and read the speech modality data (i.e., audio files) of each utterance according to the file name. Then preprocess the audio file, including pre-stress, framing, windowing, and other steps. In the audio file, the original speech signal is usually a continuous waveform representing the speaker's voice. Then, various features in the audio data are extracted through openSMILE through the feature set in the IS10 configuration file. The set contains 1582 features, which are derived from 34 basic low-level descriptors (LLDs) and their corresponding 34 incremental coefficients. Each LLD profile is applied with 21 functions (1428 features). In addition, 19 functions (152 features) are applied to the four pitch-based LLDs and their four incremental coefficient profiles, and the number of pitch onsets (pseudosyllables) and the total duration of the input (2 features) are also added. LLD includes but is not limited to acoustic features such as sound loudness, MFCC, voiced characteristics, F0 fundamental frequency, etc. Finally, by batch processing the entire dataset, the audio features of all utterances are obtained, and each utterance is represented as a 1582-dimensional feature vector.

[0096] For video files, see Figure 4 , a DenseNet model is used to extract visual features from video files (video files recording conversation scenes can be composed of multiple image frames). DenseNet (Densely Connected Convolutional Networks) is a CNN architecture. In DenseNet, each layer receives additional input from all previous layers and passes its own feature map to all subsequent layers. The grayscale images are scaled to a uniform size (48×48×1) to ensure that they can be effectively processed by the DenseNet model. The scaled and standardized images are then input into the pre-trained DenseNet model. DenseNet extracts high-level features from images through its deep convolutional network structure, which can represent the emotional content in the image. The pre-trained model has learned a large number of visual feature representations, so it can effectively identify emotion-related features from the input image. Finally, the DenseNet model outputs a 1×342-dimensional image emotion vector feature as a video feature.

[0097] In this embodiment, feature encoding is performed on the feature set to obtain the encoding features corresponding to the feature set, which may include:

[0098] Each sentence in the feature set is feature encoded through a bidirectional gated recurrent unit:

[0099]

[0100] In the formula, represents a bidirectional gated recurrent unit in text mode, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, and Represent the text features, audio features and video features of the i-th sentence respectively, and Respectively represent the encoding features corresponding to the text features, audio features, and video features of the i-th sentence, and Respectively represent the coding features corresponding to the text features, audio features, and video features of the i+1th or i-1th sentence.

[0101] In this embodiment, performing a graph convolution operation on the coding feature to obtain a graph convolution operation result may include:

[0102] According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows:

[0103]

[0104]

[0105] In the formula, h′ i represents the convolution operation result corresponding to the encoding feature of the i-th sentence, h i Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and A collection of softmax represents the first activation function, W1, W2, W3, and W4 represent trainable parameter matrices, and h j Represents the encoding features of the jth sentence in text, acoustic and visual modalities and A collection of represents the set of all sentences in the same dialogue as the i-th sentence, and d represents the dimension of the encoded features.

[0106] In this embodiment, please refer to Figure 6 In the local graph representation part, in order to capture the feature relationship of a sentence in each modality, a local graph is constructed for each modality. For a dialogue scene containing N sentences, it is represented by a directed graph as in, and ε represent nodes (a sentence is regarded as a node) and adjacency matrix respectively, and R represents the type of relationship connecting nodes. Each node in the graph is initialized by the encoding features corresponding to the node in the three modalities of text, acoustics, and vision.

[0107] For the connection relationship between nodes in the local graph, if there are M (M ≥ 2) interlocutors in the dialogue scene, the dialogue relationship between the interlocutors can be divided into the speaker itself (S1) and other people in the dialogue scene (S2) (refer to Figure 6 , in this embodiment, two people are used as an example). Based on this, four types of undirected dynamic information transmission relationships are defined:

[0108] 1. The speaker’s own utterances are self-connected (s1→s1);

[0109] 2. Self-connections between utterances of other speakers (s2→s2);

[0110] 3. The connection relationship between the speaker and other speakers (s1→s2);

[0111] 4. The connection relationship between other speakers and the speaker itself (s2→s1).

[0112] In addition, if we focus on the temporal relationship in the conversation, the relationship between the current sentence and the previous sentence (forward connection) and the relationship between the current sentence and the future sentence (backward connection) will form 8 types of relationships. And the backward connection of (s1→s2) is not symmetrical with the forward connection of (s2→s1) because they have different start and end nodes.

[0113] Based on the above-mentioned local graph structure, this embodiment performs graph convolution on the coding features through the above-mentioned graph convolution operation formula to obtain a convolution operation result, thereby converting the coding features that are not related to the interlocutor into feature vectors related to the interlocutor, and obtaining the intra-modal features (i.e., the convolution operation results) corresponding to the audio features, video features, and text features containing the interlocutor information.

[0114] In this embodiment, performing emotion recognition on the convolution operation result to obtain the first recognition result may include:

[0115] Perform emotion recognition on the convolution operation result:

[0116]

[0117] In the formula, Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and The corresponding convolution operation result, W l and W ucrepresents the trainable weight matrix, b l and b uc represents a trainable bias term, express The feature vector after being processed by the second activation function ReLU, represents the predicted probability distribution of the i-th sentence in the first recognition result in text, acoustic and visual modalities, Softmax represents the first activation function, Represents the unimodal sentiment label corresponding to the i-th sentence in text, acoustic, and visual modalities.

[0118] In this embodiment, according to the convolution operation result, by presetting a multi-head attention splicing strategy, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​to obtain multimodal features, which may include:

[0119] Map the audio features and the video features to the text space as the attention output of a single attention head:

[0120]

[0121] in:

[0122]

[0123] In the formula, H a ′、H t ′ and H v ′ represents the convolution operation results corresponding to audio features, text features and video features respectively, Q (v-t) , K (v-t) and V (v-t) represents the attention parameter obtained by mapping video features to text space, Q (a-t) , K (a-t) and V (a-t) represents the attention parameter obtained by mapping audio features to text space, and Respectively represent the trainable weight matrices;

[0124] The attention output corresponding to the audio feature and the attention output corresponding to the video feature are concatenated respectively to obtain the attention features of the audio feature and the video feature in the text space:

[0125]

[0126] In the formula, A (a-t) Represents the attention feature of the audio feature in the text space, A (v-t)represents the attention feature of the video feature in the text space, Concat represents concatenation, and N att Indicates the number of attention heads;

[0127] The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are concatenated to obtain the multimodal feature:

[0128] H m =Concat(A (a-t) ,H′ t ,A (v-t) ) (8)

[0129] In the formula, H m Represents multimodal features, H′ t The convolution operation results corresponding to the encoded features representing the text modality.

[0130] In this embodiment, refer to Figure 5 The processing flow of the preset multi-head attention splicing strategy is to use two attention modules to perform convolution operations on the input result H a ′、H t ′ and H v ′The feature conversion is performed through equations (5) and (6), so that the audio features and video features are mapped to the text space. Then, the attention output corresponding to the audio features and the attention output corresponding to the video features are spliced ​​through the multi-head attention mechanism through equation (7) to obtain the attention features of the audio features in the text space and the attention features of the video features in the text space. Finally, the convolution operation results corresponding to the attention features of the audio features in the text space, the attention features of the video features in the text space, and the encoding features of the text modality are spliced ​​to obtain the multimodal feature H. m .

[0131] In this embodiment, performing emotion recognition on the multimodal feature to obtain the second recognition result may include:

[0132] Each sentence in the sample to be tested is taken as a node, and the connection relationship between the current node and other nodes in the same conversation is identified according to the multimodal features corresponding to the node, so as to obtain the relationship features:

[0133]

[0134] In the formula, represents the relationship feature of the i-th node, represents the multimodal features corresponding to the i-th node, Represents the multimodal features corresponding to the jth node, j∈nr (i), n r (i) represents the set of all nodes in the same conversation with the i-th node, |n r (i)| is the normalization constant, θ root and θ r Represents trainable parameters;

[0135] Obtaining weight parameters between the current node and other nodes in the same conversation;

[0136] According to the relationship features corresponding to each node and the weight parameter, the global features of each node are determined:

[0137]

[0138] In the formula, g i represents the global feature corresponding to the i-th node, represents the relationship feature corresponding to the i-th node, represents the relationship feature corresponding to the jth node, j∈n(i), n(i) represents the set of all nodes in the same conversation with the i-th node, and represents the trainable weight matrix, e i,j represents the weight parameter;

[0139] The second recognition result is determined according to the relationship feature and the global feature.

[0140] In this embodiment, please refer to Figure 6 In the global graph representation part, the structural relationship of the global graph is the same as that of the local graph. There are 4 undirected connection relationships between the interlocutors, and each connection relationship has 2 types of time connections, for a total of 8 connection relationships. In this embodiment, the relationship fusion information between the current node and other nodes in the same paragraph is captured by the above formula (9), and the relationship fusion information is refined and enhanced by the above formula (10) to achieve multimodal feature fusion of different information.

[0141] Wherein, obtaining the weight parameter between the current node and other nodes in the same conversation may include:

[0142] Determine the weight parameter according to the relationship feature corresponding to each node:

[0143]

[0144] In the formula, e i,j Represents the weight parameter between node i and node j.

[0145] Wherein, determining the second recognition result according to the relationship feature and the global feature may include:

[0146] Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node:

[0147]

[0148] In the formula, e i Indicates the feature representation of the sentence corresponding to the current node;

[0149] Emotion recognition is performed according to the feature representation to obtain the second recognition result:

[0150] l i =RELU(W j w i +b j ) (13)

[0151]

[0152] Where W j and W k represents the trainable weight matrix, b j and b k represents a trainable bias term, softmax represents the first activation function, RELU represents the second activation function, l i Represents the feature vector after being processed by the second activation function RELU, Indicates the second recognition result.

[0153] In step 130, the first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested, which may include:

[0154] The first recognition result and the second recognition result are weightedly summed to obtain the emotion prediction result:

[0155]

[0156] In the formula, represents the sentiment prediction result, represents the predicted probability distribution of the i-th sentence in the text, acoustic, and visual modalities in the first recognition result, and β, γ, η, ζ represent the preset hyperparameters.

[0157] In summary, please refer to Figure 6 In practical applications, Figure 6The MMGCN (multimodal emotion recognition) model shown implements the multimodal emotion recognition method for dialogue scenarios proposed in this application. Among them, the MMGCN model includes a unimodal context editing module, a local graph module, a unimodal classification and fusion module, a global graph module and a multimodal classifier. Combined with the aforementioned multimodal emotion recognition method for dialogue scenarios, the audio files, text files and video files recording the dialogue scenes are input into the MMGCN model, and then the features of each input file are extracted through the unimodal context editing module to obtain the encoding features under each modality. and Then, the local graph module is used to perform graph convolution on the encoded features to obtain the convolution result h′ i , that is, the convolution operation results corresponding to each mode and Then, the unimodal classifier in the unimodal classification and fusion module is used to perform sentiment prediction on the convolution operation results under the three modes, and the unimodal prediction probability distribution corresponding to each sentence in the dialogue scene is obtained. That is, the first recognition result, and the unimodal emotion label representing the unimodal emotion prediction result Then, through the cross-modal fusion attention module in the single-modal classification and fusion module, based on the preset multi-head attention splicing strategy, the intra-modal features corresponding to the audio features, video features and text features are spliced ​​to obtain the multi-modal feature H m , and then extract the relational features of multimodal features through the global graph module and the global feature g i , and connect the relational features and global features through a multimodal classifier to obtain the feature representation e corresponding to each sentence in the dialogue scene i , and according to e i Determine the second recognition result Finally, the first recognition result of the unimodal classifier output And the second recognition result output by the multimodal classifier Perform weighted summation to obtain the label representing the sentiment prediction result In this way, the information in the three modalities of text, audio and video can be effectively captured, the emotional content contained in the conversation between the interlocutors can be more comprehensively represented, and based on the preset multi-head attention splicing strategy, the limitations of feature alignment and fusion between different modalities can be improved, thereby improving the accuracy of emotion recognition for dialogue scenarios.

[0158] In order to reflect the technical advantages of the embodiments of the present application, the MMGCN model is evaluated using two benchmark datasets, IEMOCAP and MELD. Both datasets are multimodal datasets that contain text, visual, and acoustic information for each sentence in each session.

[0159] The IEMOCAP dataset is an interactive emotional binary motion capture database released by the SAIL laboratory of the University of Southern California. It contains two-way conversation videos of ten different interlocutors (five men and five women), including audio-visual data with text transcription of 12 hours of video. The entire dataset has 5 chapters, each of which contains a conversation video of two interlocutors. Each video contains a binary dialogue, and each dialogue is divided into a single sentence. The emotional labels of the sentences include six categories: happy, sad, neutral, excited, frustrated, and angry.

[0160] MELD is an extension of the Emotion Lines dataset designed for emotion recognition in conversations. EmotionLines is a purely textual conversation dataset. MELD expands Emotion Lines with speech and visual modalities and has three higher-quality modality-aligned conversation data. Unlike binary conversation datasets such as IEMOCAP, MELD has more than two interlocutors in a conversation. The dataset has a total of 13,708 sentences, 1,433 conversations, and a total of 304 different interlocutors. Each sentence is annotated with an emotion label, emotion polarity, and interlocutor information. Each sentence in the conversation is annotated with one of the seven emotion labels, including anger, disgust, fear, happiness, neutrality, sadness, and surprise.

[0161] This application sets up a comparative analysis under different modalities, showing the performance comparison of the MMGCN model proposed in this application under different modal settings on two benchmark datasets. As can be seen from Table 1 below, the best single-modal performance is achieved on the text modality, while the worst is achieved on the visual modality. Adding acoustic and visual modalities can bring additional performance improvements over the text modality.

[0162] Table 1 Performance of MMGCN under different multi-modal settings

[0163]

[0164] The comparative analysis of the MMGCN model proposed in this application and other existing emotion recognition models is shown in Table 2 below. The MMGCN model achieved the best results on two datasets of multimodal emotion recognition. On the IEMOCAP dataset, the MMGCN model achieved an accuracy of 70.12% and an F1 score of 70.22%. On MELD, the MMGCN model also surpassed all other models, improving F1 by 5.72% over MM-DFN, and achieving an accuracy of 66.21% and a weighted F1 score of 65.18%.

[0165] Table 2 Model performance comparison

[0166]

[0167]

[0168] Through the description of the above implementation methods, technical personnel in this field can clearly understand that the present application can be implemented by hardware, and can also be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0169] In the embodiments provided by the present application, it should be understood that the disclosed method can also be implemented in other ways. The method embodiments described above are merely schematic, for example, the flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and a part of the module, a program segment or a code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or the flow chart, and the combination of the boxes in the block diagram and / or the flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0170] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal emotion recognition method for dialogue scenarios, characterized in that: The method comprises: Acquire a sample to be tested, wherein the sample to be tested includes an audio file, a video file, and a text file recording the same conversation scene; By presetting a multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result and a second recognition result are obtained respectively, wherein the first recognition result includes a single-modal prediction probability distribution corresponding to the sample to be tested, and the second recognition result includes a multimodal fusion prediction probability distribution corresponding to the sample to be tested; The first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested.

2. The method according to claim 1, characterized in that By presetting a multimodal emotion recognition strategy, emotion recognition is performed on the sample to be tested, and a first recognition result and a second recognition result are obtained respectively, including: Performing feature extraction on the sample to be tested to obtain audio features corresponding to the audio file, video features corresponding to the video file, and text features corresponding to the text file as a feature set; Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set; Performing a graph convolution operation on the encoded features to obtain a graph convolution operation result, wherein the graph convolution operation result includes intra-modal features corresponding to the audio features, the video features, and the text features, respectively; Performing emotion recognition on the convolution operation result to obtain the first recognition result, wherein the first recognition result includes the single-modal prediction probability distribution corresponding to different modalities of each sentence in the sample to be tested; According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​respectively by a preset multi-head attention splicing strategy to obtain a multi-modal feature; Emotion recognition is performed on the multimodal features to obtain the second recognition result, where the second recognition result includes the multimodal fusion prediction probability distribution corresponding to each sentence in the sample to be tested.

3. The method according to claim 2, characterized in that Performing feature encoding on the feature set to obtain encoding features corresponding to the feature set includes: Each sentence in the feature set is feature encoded through a bidirectional gated recurrent unit: In the formula, represents a bidirectional gated recurrent unit in text mode, represents a bidirectional gated recurrent unit located in the acoustic mode, represents a bidirectional gated recurrent unit in the visual modality, and Represent the text features, audio features and video features of the i-th sentence respectively, and Respectively represent the encoding features corresponding to the text features, audio features, and video features of the i-th sentence, and Respectively represent the coding features corresponding to the text features, audio features, and video features of the i+1th or i-1th sentence.

4. The method according to claim 2, characterized in that: Performing a graph convolution operation on the encoding feature to obtain a graph convolution operation result, including: According to the coding feature, a graph convolution operation is performed on the coding feature to obtain a convolution operation result, wherein the formula of the graph convolution operation is as follows: In the formula, h′ i represents the convolution operation result corresponding to the encoding feature of the i-th sentence, h i Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and A collection of softmax represents the first activation function, W1, W2, W3, and W4 represent trainable parameter matrices, and h j Represents the encoding features of the jth sentence in text, acoustic and visual modalities and A collection of represents the set of all sentences in the same dialogue as the i-th sentence, and d represents the dimension of the encoded features.

5. The method according to claim 2, characterized in that: Performing emotion recognition on the convolution operation result to obtain the first recognition result includes: Perform emotion recognition on the convolution operation result: In the formula, Represents the encoding features of the i-th sentence in text, acoustic and visual modalities and The corresponding convolution operation result, W l and W uc represents the trainable weight matrix, b l and b uc represents a trainable bias term, express The feature vector after being processed by the second activation function ReLU, represents the predicted probability distribution of the i-th sentence in the first recognition result in text, acoustic and visual modalities, Softmax represents the first activation function, Represents the unimodal sentiment label corresponding to the i-th sentence in text, acoustic, and visual modalities.

6. The method according to claim 2, characterized in that According to the convolution operation result, the intra-modal features corresponding to the audio features, the video features, and the text features are spliced ​​by presetting a multi-head attention splicing strategy to obtain multimodal features, including: Map the audio features and the video features to the text space as the attention output of a single attention head: in: In the formula, H a ′、H t ′ and H v ′ represents the convolution operation results corresponding to audio features, text features and video features respectively, Q (v -t) , K (v-t) and V (v-t) represents the attention parameter obtained by mapping video features to text space, Q (a-t) , K (a-t) and V (a-t) represents the attention parameter obtained by mapping audio features to text space, and Respectively represent the trainable weight matrices; The attention output corresponding to the audio feature and the attention output corresponding to the video feature are concatenated respectively to obtain the attention features of the audio feature and the video feature in the text space: In the formula, A (a-t) Represents the attention feature of the audio feature in the text space, A (v-t) represents the attention feature of the video feature in the text space, Concat represents concatenation, and N att Indicates the number of attention heads; The attention feature of the audio feature in the text space, the attention feature of the video feature in the text space, and the convolution operation results corresponding to the encoding feature of the text modality are concatenated to obtain the multimodal feature: H m =Concat(A (a-t) ,H′ t ,A (v-t) ) In the formula, H m Represents multimodal features, H′ t The convolution operation results corresponding to the encoded features representing the text modality.

7. The method according to claim 2, characterized in that: Performing emotion recognition on the multimodal features to obtain the second recognition result includes: Each sentence in the sample to be tested is taken as a node, and the connection relationship between the current node and other nodes in the same conversation is identified according to the multimodal features corresponding to the node, so as to obtain the relationship features: In the formula, represents the relationship feature of the i-th node, represents the multimodal features corresponding to the i-th node, Represents the multimodal features corresponding to the jth node, j∈n r (i), n r (i) represents the set of all nodes in the same conversation with the i-th node, |n r (i)| is the normalization constant, θ root and θ r Represents trainable parameters; Obtaining weight parameters between the current node and other nodes in the same conversation; According to the relationship features corresponding to each node and the weight parameter, the global features of each node are determined: In the formula, g i represents the global feature corresponding to the i-th node, represents the relationship feature corresponding to the i-th node, represents the relationship feature corresponding to the jth node, j∈n(i), n(i) represents the set of all nodes in the same conversation with the i-th node, and represents the trainable weight matrix, e i,j represents the weight parameter; The second recognition result is determined according to the relationship feature and the global feature.

8. The method according to claim 7, characterized in that Obtaining weight parameters between the current node and other nodes in the same conversation, including: Determine the weight parameter according to the relationship feature corresponding to each node: In the formula, e i,j Represents the weight parameter between node i and node j.

9. The method according to claim 7, characterized in that: Determining the second recognition result according to the relationship feature and the global feature includes: Connect the relationship feature and the global feature to obtain the feature representation of the sentence corresponding to the current node: e i =[h′ i m ,g i ] In the formula, e i Indicates the feature representation of the sentence corresponding to the current node; Emotion recognition is performed according to the feature representation to obtain the second recognition result: l i =RELU(W j have been i +b j ) Where W j and W k represents the trainable weight matrix, b j and b k represents a trainable bias term, softmax represents the first activation function, RELU represents the second activation function, l i Represents the feature vector after being processed by the second activation function RELU, Indicates the second recognition result.

10. The method according to claim 1, characterized in that The first recognition result and the second recognition result are concatenated to obtain an emotion prediction result representing the emotion state of the current sentence in the sample to be tested, including: The first recognition result and the second recognition result are weightedly summed to obtain the emotion prediction result: In the formula, represents the sentiment prediction result, represents the predicted probability distribution of the i-th sentence in the text, acoustic and visual modalities in the first recognition result, β, γ, η, Represents the preset hyperparameters.

Images