A multimodal conversation emotion recognition method based on multi-model voting

Abstract

The present invention provides a multimodal dialogue emotion recognition method based on multi-model voting. First, multimodal data generated by the speech of at least one speaker is obtained, and emotion classification tasks of three modes are constructed for text data, audio data and image data respectively, and the first emotion classification is performed; the multimodal data is fused by a multi-head attention mechanism and the second emotion classification is performed; the multimodal emotion feature vector is fused with temporal context information and the third emotion classification is performed; finally, the three emotion classification results are hard voted, and the emotion category with the largest number of votes for each speaker is taken as the final emotion classification result, so as to complete the multimodal dialogue emotion recognition; the present invention avoids emotion interference by optimizing the multimodal interaction mode, and at the same time models the interaction between historical dialogues and speakers, and mines the emotion features contained in each mode in a more detailed manner, so as to enhance the accuracy and robustness of emotion classification.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and dialogue emotion classification, and more specifically, to a multimodal dialogue emotion recognition method based on multi-model voting. Background Art

[0002] Conversational sentiment classification (ERC) has long been a research direction that has attracted much attention in the fields of multimodal classification and natural language processing (NLP). In daily human communication, identifying and tracking the emotional state of conversation participants is crucial to the progress of human-computer interaction, conversation analysis, and video understanding, and has a wide range of potential applications. With the development of streaming services, multimodal conversational sentiment recognition has demonstrated a wide range of application scenarios and important significance in the fields of intelligent customer service, social media analysis, emotion-driven content recommendation, sentiment analysis research, and emotion-driven human-computer interaction. Taking the emotion-driven human-computer interaction system as an example, this technology can identify and analyze the emotional state expressed by users during the conversation, such as anger, satisfaction, frustration, etc. The system can adaptively adjust the interaction mode, tone, and feedback according to the different emotional needs of users, so as to interact with users emotionally and provide more personalized and humane services.

[0003] Traditional multimodal emotion recognition methods have some limitations and cannot effectively solve problems such as opposite emotions expressed between modalities, multiple emotions expressed in a single modality, and modal influences between interlocutors, which affects emotion discrimination.

[0004] Emotion recognition in multimodal conversations usually has the following difficulties: First, in multimodal conversation scenarios, unlike traditional single-sentence multimodal emotion recognition, multimodal emotion recognition faces more challenging problems. This is because there are multiple factors in the conversation that affect the speaker's emotional state, including multimodal context, interlocutor stimulation, own emotional inertia, conversation scenarios, and personality traits. Therefore, it is necessary to model and predict emotions from different modalities and angles for multimodal conversations; second, there may be conflicts between the emotions expressed in different modalities, such as a person smiling but expressing sadness, so special attention should be paid to handling this situation during modal interaction; third, a single modality can also present multiple emotions, so a more fine-grained emotion classification method is needed; fourth, different speakers and interlocutors in the conversation may influence each other, and when the interlocutor does not speak, the speaker's conversation emotion may be stimulated by the interlocutor's expression. Therefore, in the modeling process, the consideration of interactive information is not limited to the speaker's own video, audio, and text features, but also the corresponding video features of the interlocutor. Fifth, when classifying conversation emotions, it is necessary to analyze the current speaker's emotions in combination with the historical conversation content. Even if the current conversation content does not have an obvious emotional tendency, the emotional change trend can be analyzed from the historical multimodal conversation data to better make emotional judgments.

[0005] The prior art discloses a multimodal classification method and system based on dynamic context representation and modal fusion, which solves the problem that the features of each modality are not fully analyzed and not processed in a targeted manner according to their characteristics. The method performs global context representation, local context representation and direct mapping representation on the features of each modality respectively, and then fuses the above representations according to a dynamic path selection method to obtain the initial fusion features of each modality; the initial fusion features of all modalities are respectively subjected to full fusion, partial fusion and biased fusion processing to obtain full fusion results, partial fusion results and biased fusion results, and then are fused through a dynamic path selection method to obtain the final fusion features for The multimodal fusion features of classification improve the accuracy of the final recognition task category; although the method in the prior art can solve the problem that the modal interaction form is too simple, and the characteristics of each modality are not fully analyzed and not processed in a targeted manner according to its characteristics, by distinguishing and processing the inconsistent information amounts of different modalities in the multimodal process, the noise caused by the modality with less information amount in the modal fusion process is reduced, but usually a conversation content contains multiple emotions. This method cannot mine the emotional features contained in different modalities from a finer granularity, and when the emotions expressed by different modalities are inconsistent, it will interfere with the modal fusion. Summary of the invention

[0006] In order to overcome the defects of the above-mentioned prior art in that the modal is not mined in a fine-grained manner during conversation emotion recognition, the modal interaction is easily disturbed and the recognition accuracy is low, the present invention provides a multimodal conversation emotion recognition method based on multi-model voting, which avoids emotional interference by optimizing the multimodal interaction mode, and at the same time models the interaction between historical conversations and speakers, and mines the emotional features contained in each modality in a more detailed manner, thereby enhancing the accuracy and robustness of emotion classification.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] A multimodal dialogue emotion recognition method based on multi-model voting includes the following steps:

[0009] S1: Acquire multimodal data generated by a conversation between at least one speaker; the multimodal data includes text data, audio data, and image data; and each modality of data includes at least one emotion category to be identified;

[0010] S2: Inputting the text data, audio data and picture data into a preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtaining text features, audio features and picture features respectively;

[0011] S3: inputting the text features, audio features and picture features into a preset text sentiment classifier, audio sentiment classifier and picture sentiment classifier respectively to perform a first sentiment classification, and obtaining a text sentiment classification result, an audio sentiment classification result and a picture sentiment classification result respectively;

[0012] S4: Calculate the penalty factor corresponding to each modality according to the text sentiment classification results, the audio sentiment classification results, and the image sentiment classification results; multiply the text features, the audio features, and the image features by the penalty factor corresponding to each modality to obtain the text down-weighting vector, the audio down-weighting vector, and the image down-weighting vector respectively;

[0013] S5: Input the text down-weighted vector, the audio down-weighted vector, and the image down-weighted vector into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal emotion feature vector;

[0014] S6: input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result;

[0015] S7: decomposing the multimodal emotion feature vector into a plurality of multimodal emotion feature sub-vectors, and rejoining all the multimodal emotion feature sub-vectors in time sequence to obtain an emotion feature vector integrating time sequence features;

[0016] S8: Input the sentiment feature vector fused with the temporal features into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result;

[0017] S9: Hard voting is performed on the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and temporal context interaction sentiment classification results. The sentiment category with the largest number of votes for each speaker is taken as the final sentiment classification result to complete multimodal dialogue sentiment recognition.

[0018] Preferably, the multimodal data in step S1 is acquired by respectively extracting text data, audio data and image data from conversation video data of at least one preset speaker to acquire the multimodal data.

[0019] Preferably, the step S1 further includes preprocessing the acquired audio data and image data, and the specific method is as follows:

[0020] The audio data is sequentially subjected to sampling rate adjustment, denoising and audio gain adjustment, and an audio segment is extracted using a sliding window to complete preprocessing of the audio data;

[0021] The image data is deduplicated, specifically:

[0022] S1.1: performing optical flow calculation on two pictures of adjacent frames of the conversation video data using an optical flow algorithm to obtain an optical flow calculation result;

[0023] S1.2: Calculate the change range of two pictures of adjacent frames according to the optical flow calculation results, and determine whether the change range is greater than a preset threshold. If so, retain the two pictures of the adjacent frames in the deduplication result set and execute step S1.3; otherwise, directly execute step S1.3;

[0024] S1.3: Starting from the first frame of the conversation video data, repeat steps S1.1~S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed, and save the deduplication result set obtained from the last update as the deduplication picture data to complete the preprocessing of the picture data.

[0025] Preferably, the optical flow algorithm in step S1.1 is any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm.

[0026] Preferably, the change amplitude in step S1.2 is specifically the displacement vector size or similarity measure of each pixel in two pictures of adjacent frames; the similarity measure includes Euclidean distance and angle change.

[0027] Preferably, in step S2, the preset text encoder is specifically a Transformer encoder, and the Transformer encoder is specifically any one of a BERT model, a MacBERT model, a RoBERTa model, and an ERNIE model;

[0028] The preset audio encoder is specifically a HuBERT model;

[0029] The preset picture encoder is specifically a ViT model.

[0030] Preferably, the text sentiment classifier, audio sentiment classifier and picture sentiment classifier preset in step S3, and the multimodal sentiment classifier preset in step S6 are all CRF classifiers.

[0031] Preferably, in step S4, the specific method for calculating the penalty factor corresponding to each mode is:

[0032] According to the text sentiment classification results, audio sentiment classification results and image sentiment classification results, the types of sentiment categories n are obtained, and the total number of occurrences of each sentiment category is counted respectively, and the proportion of the i-th sentiment category is calculated. , specifically:

[0033]

[0034] in, is the number of occurrences of the i-th emotion category;

[0035] For the classification results of each modality, the proportions of all the emotion categories it contains are added together, and the sum is used as the penalty factor of the corresponding modality.

[0036] Preferably, the specific method of step S7 is:

[0037] Decomposing the multimodal emotion feature vector into a number of multimodal emotion feature sub-vectors according to different speakers, each multimodal emotion feature sub-vector corresponds to each speaker one by one;

[0038] The multimodal emotion feature sub-vector includes a text down-weighting sub-vector, an audio down-weighting sub-vector and a picture down-weighting sub-vector;

[0039] The image down-weighted sub-vector corresponding to each speaker is fused with the multimodal emotion feature sub-vectors corresponding to all other speakers to obtain the fused multimodal emotion feature sub-vector corresponding to each speaker;

[0040] The fused multimodal emotion feature sub-vectors corresponding to all speakers are reconnected according to the speaking order of different speakers to obtain the emotion feature vector fused with the temporal features.

[0041] Preferably, the bidirectional RNN classifier in step S8 is specifically:

[0042] The bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier having the same structure and arranged in parallel;

[0043] The forward RNN classifier and the backward RNN classifier both include a plurality of RNN classification layers connected in sequence;

[0044] All RNN classification layers have the same structure, which is any one of the LSTM network, GRU network, and PQRNN network.

[0045] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0046] The present invention provides a multimodal dialogue emotion recognition method based on multi-model voting, which comprises the following steps: firstly, obtaining multimodal data generated by a dialogue of at least one speaker; respectively inputting text data, audio data and picture data into a preset text encoder, audio encoder and picture encoder for feature extraction, and respectively obtaining text features, audio features and picture features; respectively inputting text features, audio features and picture features into a preset text emotion classifier, audio emotion classifier and picture emotion classifier for first emotion classification, and respectively obtaining text emotion classification results, audio emotion classification results and picture emotion classification results; respectively calculating a penalty factor corresponding to each mode according to the text emotion classification results, audio emotion classification results and picture emotion classification results; respectively multiplying the text features, audio features and picture features by the penalty factor corresponding to each mode, and respectively obtaining a text reduction weight vector, an audio reduction weight vector and a picture reduction weight vector; and respectively multiplying the text reduction weight vector, the audio reduction weight vector and the picture reduction weight vector. The reduced-weight vectors are jointly input into the preset multi-head attention layer for fusion interaction of multimodal features to obtain a multimodal sentiment feature vector; the multimodal sentiment feature vector is input into the preset multimodal sentiment classifier for a second sentiment classification to obtain a multimodal fusion sentiment classification result; the multimodal sentiment feature vector is decomposed into several multimodal sentiment feature sub-vectors, and all multimodal sentiment feature sub-vectors are reassembled in time sequence to obtain a sentiment feature vector fused with time sequence features; the sentiment feature vector fused with time sequence features is input into the trained bidirectional RNN classifier for a third sentiment classification to obtain a time sequence context interaction sentiment classification result; finally, the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and time sequence context interaction sentiment classification results are jointly hard voted, and the sentiment category with the largest number of votes for each speaker is taken as its final sentiment classification result to complete multimodal dialogue sentiment recognition;

[0047] The present invention can solve the problem that the existing technology cannot recognize multiple different emotions expressed in different modalities by constructing fine-grained sentiment classification tasks for each modality. Through this training task of extracting sentiment fragments, which is more difficult than ordinary sentiment classification training, the sensitivity of the classification model to the location where the modal sentiment features appear is improved, thereby improving the judgment accuracy of the model; secondly, the present invention uses the fine-grained sentiment classification tasks for each modality to guide the biased fusion interaction of each modality, thereby avoiding the influence of errors and noise of modal sentiment information on the interaction results; in addition, the present invention also provides a temporal context interaction method for multimodal data, while combining context information and the features of mutual influence between different speakers in a conversation, so as to better understand the emotional content of the conversation; when one speaker listens and another speaker speaks, the present invention also considers the influence of the listener's expression and picture features on the current speaker's emotions, and can more accurately identify the emotional content in the conversation, effectively enhancing the accuracy and robustness of sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of a multimodal dialogue emotion recognition method based on multi-model voting provided in Example 1.

[0049] Figure 2 This is a schematic diagram of the text data emotion fragments and emotion labels provided in Example 2.

[0050] Figure 3 Schematic diagram of the text encoder and text sentiment classifier provided in Example 2.

[0051] Figure 4 Schematic diagram of the audio encoder and audio emotion classifier provided in Example 2.

[0052] Figure 5 This is a schematic diagram of the image data deduplication operation provided in Example 2.

[0053] Figure 6 This is a schematic diagram of the third emotion classification provided in Example 2. DETAILED DESCRIPTION

[0054] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present application;

[0055] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0056] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0057] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a multimodal dialogue emotion recognition method based on multi-model voting, comprising the following steps:

[0060] S1: Acquire multimodal data generated by a conversation between at least one speaker; the multimodal data includes text data, audio data, and image data; and each modality of data includes at least one emotion category to be identified;

[0061] S2: Inputting the text data, audio data and picture data into a preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtaining text features, audio features and picture features respectively;

[0062] S3: inputting the text features, audio features and picture features into a preset text sentiment classifier, audio sentiment classifier and picture sentiment classifier respectively to perform a first sentiment classification, and obtaining a text sentiment classification result, an audio sentiment classification result and a picture sentiment classification result respectively;

[0063] S4: Calculate the penalty factor corresponding to each modality according to the text sentiment classification results, the audio sentiment classification results, and the image sentiment classification results; multiply the text features, the audio features, and the image features by the penalty factor corresponding to each modality to obtain the text down-weighting vector, the audio down-weighting vector, and the image down-weighting vector respectively;

[0064] S5: Input the text down-weighted vector, the audio down-weighted vector, and the image down-weighted vector into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal emotional feature vector;

[0065] S6: input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result;

[0066] S7: decomposing the multimodal emotion feature vector into a plurality of multimodal emotion feature sub-vectors, and rejoining all the multimodal emotion feature sub-vectors in time sequence to obtain an emotion feature vector integrating time sequence features;

[0067] S8: Input the sentiment feature vector fused with the temporal features into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result;

[0068] S9: Hard voting is performed on the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and temporal context interaction sentiment classification results. The sentiment category with the largest number of votes for each speaker is taken as the final sentiment classification result to complete multimodal dialogue sentiment recognition.

[0069] In the specific implementation process, firstly, multimodal data generated by a conversation of at least one speaker is obtained; the text data, audio data and image data are respectively input into a preset text encoder, audio encoder and image encoder for feature extraction, and text features, audio features and image features are respectively obtained;

[0070] Input text features, audio features and picture features into preset text sentiment classifiers, audio sentiment classifiers and picture sentiment classifiers respectively for first sentiment classification, and obtain text sentiment classification results, audio sentiment classification results and picture sentiment classification results respectively;

[0071] Calculate the penalty factor corresponding to each modality according to the text sentiment classification results, audio sentiment classification results, and image sentiment classification results; multiply the text features, audio features, and image features by the penalty factor corresponding to each modality to obtain the text down-weighting vector, audio down-weighting vector, and image down-weighting vector, respectively;

[0072] The text down-weighted vector, audio down-weighted vector and image down-weighted vector are input into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal emotional feature vector.

[0073] Input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result;

[0074] Decomposing the multimodal emotion feature vector into several multimodal emotion feature sub-vectors, and rejoining all the multimodal emotion feature sub-vectors in time sequence to obtain an emotion feature vector integrating time sequence features;

[0075] The sentiment feature vector fused with the temporal features is input into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result;

[0076] Finally, the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and temporal context interaction sentiment classification results are jointly hard voted, and the sentiment category with the most votes for each speaker is taken as its final sentiment classification result to complete multimodal dialogue sentiment recognition;

[0077] This method avoids emotional interference by optimizing the multimodal interaction mode, and at the same time models the interaction between historical dialogues and speakers, mining the emotional features contained in each modality in a more detailed way, which can enhance the accuracy and robustness of sentiment classification.

[0078] Example 2

[0079] This embodiment provides a multimodal dialogue emotion recognition method based on multi-model voting, including the following steps:

[0080] S1: Acquire multimodal data generated by a conversation between at least one speaker; the multimodal data includes text data, audio data, and image data; and each modality of data includes at least one emotion category to be identified;

[0081] S2: Inputting the text data, audio data and picture data into a preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtaining text features, audio features and picture features respectively;

[0082] S3: inputting the text features, audio features and picture features into a preset text sentiment classifier, audio sentiment classifier and picture sentiment classifier respectively to perform a first sentiment classification, and obtaining a text sentiment classification result, an audio sentiment classification result and a picture sentiment classification result respectively;

[0083] S4: Calculate the penalty factor corresponding to each modality according to the text sentiment classification results, the audio sentiment classification results, and the image sentiment classification results; multiply the text features, the audio features, and the image features by the penalty factor corresponding to each modality to obtain the text down-weighting vector, the audio down-weighting vector, and the image down-weighting vector respectively;

[0084] S5: Input the text down-weighted vector, the audio down-weighted vector, and the image down-weighted vector into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal emotional feature vector;

[0085] S6: input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result;

[0086] S7: decomposing the multimodal emotion feature vector into a plurality of multimodal emotion feature sub-vectors, and rejoining all the multimodal emotion feature sub-vectors in time sequence to obtain an emotion feature vector integrating time sequence features;

[0087] S8: Input the sentiment feature vector fused with the temporal features into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result;

[0088] S9: Hard voting is performed on the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results, and temporal context interaction sentiment classification results. The sentiment category with the most votes for each speaker is taken as the final sentiment classification result to complete multimodal dialogue sentiment recognition.

[0089] The multimodal data in step S1 is obtained by extracting text data, audio data and image data from the conversation video data of at least one preset speaker to obtain the multimodal data;

[0090] The step S1 also includes preprocessing the acquired audio data and image data, and the specific method is as follows:

[0091] The audio data is sequentially subjected to sampling rate adjustment, denoising and audio gain adjustment, and an audio segment is extracted using a sliding window to complete preprocessing of the audio data;

[0092] The image data is deduplicated, specifically:

[0093] S1.1: performing optical flow calculation on two pictures of adjacent frames of the conversation video data using an optical flow algorithm to obtain an optical flow calculation result;

[0094] S1.2: Calculate the change range of two pictures of adjacent frames according to the optical flow calculation results, and determine whether the change range is greater than a preset threshold. If so, retain the two pictures of the adjacent frames in the deduplication result set and execute step S1.3; otherwise, directly execute step S1.3;

[0095] S1.3: starting from the first frame of the conversation video data, repeating steps S1.1 to S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed, and saving the deduplication result set obtained by the last update as the deduplication picture data, thereby completing the preprocessing of the picture data;

[0096] The optical flow algorithm in step S1.1 is specifically any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm;

[0097] The change amplitude in step S1.2 is specifically the displacement vector size or similarity measure of each pixel in two pictures of adjacent frames; the similarity measure includes Euclidean distance and angle change;

[0098] In the step S2, the preset text encoder is specifically a Transformer encoder, and the Transformer encoder is specifically any one of a BERT model, a MacBERT model, a RoBERTa model, and an ERNIE model;

[0099] The preset audio encoder is specifically a HuBERT model;

[0100] The preset picture encoder is specifically a ViT model;

[0101] The text sentiment classifier, audio sentiment classifier and picture sentiment classifier preset in step S3, and the multimodal sentiment classifier preset in step S6 are all CRF classifiers;

[0102] In step S4, the specific method for calculating the penalty factor corresponding to each mode is:

[0103] According to the text sentiment classification results, audio sentiment classification results and image sentiment classification results, the types of sentiment categories n are obtained, and the total number of occurrences of each sentiment category is counted respectively, and the proportion of the i-th sentiment category is calculated. , specifically:

[0104]

[0105] in, is the number of occurrences of the i-th emotion category;

[0106] For the classification results of each modality, add up the proportions of all the emotion categories it contains, and use the sum as the penalty factor of the corresponding modality;

[0107] The specific method of step S7 is:

[0108] Decomposing the multimodal emotion feature vector into a number of multimodal emotion feature sub-vectors according to different speakers, each multimodal emotion feature sub-vector corresponds to each speaker one by one;

[0109] The multimodal emotion feature sub-vector includes a text down-weighting sub-vector, an audio down-weighting sub-vector and a picture down-weighting sub-vector;

[0110] The image down-weighted sub-vector corresponding to each speaker is fused with the multimodal emotion feature sub-vectors corresponding to all other speakers to obtain the fused multimodal emotion feature sub-vector corresponding to each speaker;

[0111] The fused multimodal emotion feature sub-vectors corresponding to all speakers are reconnected according to the speaking order of different speakers to obtain the emotion feature vector fused with the temporal features;

[0112] The bidirectional RNN classifier in step S8 is specifically:

[0113] The bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier having the same structure and arranged in parallel;

[0114] The forward RNN classifier and the backward RNN classifier both include a plurality of RNN classification layers connected in sequence;

[0115] All RNN classification layers have the same structure, which is any one of the LSTM network, GRU network, and PQRNN network.

[0116] In the specific implementation process, firstly, text data, audio data and image data are extracted from the conversation video data of at least one preset speaker to obtain multimodal data; each modal data includes one or more emotion categories such as happiness, frustration, calmness, joy, surprise, sadness, disgust, anger and fear, and the emotion categories can also be divided into positive and negative;

[0117] In order to solve the problem that the existing technology cannot identify multiple different emotions expressed in different modalities, this embodiment first designs fine-grained emotion classification tasks for the data of the three modalities and performs the first emotion classification, specifically:

[0118] 1) For text data, first extract the text data separately and manually annotate the emotional text fragments and emotional labels; for example, Figure 2 As shown in the figure, there are two emotional segments in the text "I feel quite happy, even though you keep making me cry", corresponding to the emotional labels of "happy (pos)" and "frustrated (neg)" respectively;

[0119] like Figure 3 As shown, the Transformer encoder is then used to extract text features, and the CRF classifier is used to learn the dependency relationship between tags and sentiment classification;

[0120] Sentiment classification of text includes two levels of judgment: one is to judge which words belong to the beginning (B), middle (I) or end (E) part of the sentiment text segment, or are not sentiment segments (O); the other is to judge which sentiment type (pos or neg) the sentiment segment belongs to;

[0121] In this embodiment, the Transformer encoder is specifically any one of BERT, MacBERT, RoBERTa and ERNIE models;

[0122] To train a model using a text dataset, you need to define a loss function and optimizer, and use the back-propagation algorithm to update the model parameters. During the training process, batch gradient descent or other optimization techniques are usually used. Based on the performance of the model on the validation set, adjust the hyperparameters, such as the learning rate, batch size, number of Transformer layers, and number of hidden units. You can use methods such as cross-validation or grid search to tune the hyperparameters.

[0123] Then, the trained model is used to perform the first sentiment classification on the input text data to obtain the text sentiment classification results, specifically: text segments containing emotional states and corresponding emotional categories;

[0124] Through the text emotion segment recognition and classification tasks, the model can focus more on identifying segments containing emotional information in the conversation text. At the same time, when there are multiple emotions in the conversation text, the model can identify them more accurately.

[0125] 2) For audio data, collect a data set containing audio data and corresponding emotion labels, and pre-process the audio signal to be identified, such as adjusting the sampling rate, denoising, and audio gain. Set the size and step size of the sliding window, and perform sliding window processing on the audio signal. Starting from the beginning of the audio, move the sliding window in sequence according to the set step size. Extract the corresponding audio segment in each window.

[0126] The size of the sliding window determines the duration of each audio segment, while the step size determines the degree of overlap between adjacent windows;

[0127] The extracted audio clips are labeled with corresponding emotional tags to ensure that each audio clip has an associated emotional tag. The beginning of each audio clip is represented by the symbol "B", the content is represented by the symbol "I", the end is represented by the symbol "E", and the non-emotional audio clip is represented by the symbol "O". In addition, the audio clips are also labeled according to the emotional category, such as "pos" for positive emotions and "neg" for negative emotions;

[0128] like Figure 4 As shown in the figure, the preprocessed audio dataset is data-augmented and then input into the HuBERT model + CRF classifier for training. The HuBERT model is used to extract the context information of the input audio sequence, while the CRF classifier is used to learn the dependencies between labels. The output feature representation of the HuBERT model is input into the CRF classifier to output the emotion type of each audio clip. A supervised learning algorithm is used for training. During the training process, the model parameters are optimized by minimizing the loss function (such as cross entropy loss), and the back-propagation algorithm is used to calculate the gradient and update the model parameters.

[0129] This embodiment also sets an emotion probability threshold parameter. For each audio clip, it is determined whether the predicted emotion category or emotion score exceeds the set emotion probability threshold parameter. If it exceeds the threshold, the time step is determined to be a clip containing emotion.

[0130] In addition, for the time step determined to contain emotion, continuous frame analysis is required. This embodiment sets a continuous frame threshold, which indicates how many consecutive time steps are determined to contain emotion before the audio segment is finally confirmed as an emotion segment.

[0131] Then, the trained HuBERT model + CRF classifier is used to perform the first sentiment classification on the input audio data to obtain the audio sentiment classification results;

[0132] 3) For image data, we first divide the dialogue video data into frames to extract feature information that expresses emotions. The feature information is mainly concentrated on the face.

[0133] Then, the image data is deduplicated, specifically:

[0134] S1.1: performing optical flow calculation on two pictures of adjacent frames of the conversation video data using an optical flow algorithm to obtain an optical flow calculation result; the optical flow algorithm can estimate the pixel displacement between adjacent frames and capture inter-frame motion information, specifically any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm;

[0135] S1.2: Calculate the change range of two pictures in adjacent frames based on the optical flow calculation results. The change range is the displacement vector size or similarity measure (such as Euclidean distance and angle change) of each pixel.

[0136] Determine whether the change amplitude is greater than a preset threshold. If so, retain the two pictures of the adjacent frames in the deduplication result set and execute step S1.3; otherwise, directly execute step S1.3;

[0137] S1.3: starting from the first frame of the conversation video data, repeating steps S1.1 to S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed, and saving the deduplication result set obtained by the last update as the deduplication picture data, thereby completing the preprocessing of the picture data;

[0138] Keep the frames with larger changes and discard the frames with smaller changes, such as Figure 5 As shown, these frames cover the speaker's change from angry to happy state, and after deduplication, the images representing the "happy" and "angry" emotion categories are retained;

[0139] The preprocessed image data is then used to train the model, using the ViT model + CRF classifier to extract image features and classification. The ViT model divides the image into image blocks (or image patches) and then rearranges them into sequences to process the image. Common loss functions used in training include smooth L1 loss for bounding box regression and cross entropy loss for classification tasks. During training, the model parameters are optimized by minimizing the loss function. Stochastic gradient descent (SGD) or other optimization algorithms can be used to update the parameters.

[0140] Finally, the trained ViT model + CRF classifier is used to perform the first emotion recognition of the image data to obtain the emotion classification result of the image data, that is, the emotion category corresponding to the face area;

[0141] After the first sentiment classification, in order to avoid the influence of noise in multimodal sentiment classification, this embodiment also performs a second sentiment classification, specifically:

[0142] According to the text sentiment classification results, audio sentiment classification results and image sentiment classification results, the penalty factor corresponding to each modality is calculated respectively. The specific method is as follows:

[0143] According to the text sentiment classification results, audio sentiment classification results and image sentiment classification results, the types of sentiment categories n are obtained, and the total number of occurrences of each sentiment category is counted respectively, and the proportion of the i-th sentiment category is calculated. , specifically:

[0144]

[0145] in, is the number of occurrences of the i-th emotion category;

[0146] For the classification results of each modality, add up the proportions of all the emotion categories it contains, and use the sum as the penalty factor of the corresponding modality;

[0147] Multiply the text features, audio features, and image features by the penalty factors corresponding to each modality to obtain the text down-weighting vector, audio down-weighting vector, and image down-weighting vector, respectively;

[0148] The text down-weighted vector, audio down-weighted vector, and image down-weighted vector are input into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal sentiment feature vector. Through the multi-head attention mechanism, the model can automatically focus on the parts of different modalities that have an important impact on sentiment classification.

[0149] Input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result;

[0150] After the second sentiment classification, this embodiment also considers the mutual influence between different speakers in the conversation and performs a third sentiment classification, specifically:

[0151] Firstly, the multimodal emotion feature vector is decomposed into several multimodal emotion feature sub-vectors according to different speakers, and each multimodal emotion feature sub-vector corresponds to each speaker one by one.

[0152] The multimodal emotion feature sub-vector includes a text down-weighting sub-vector, an audio down-weighting sub-vector and a picture down-weighting sub-vector;

[0153] like Figure 6 As shown, suppose there are two speakers, speakers A and B, who have the following conversation in chronological order:

[0154] A: (Neutral) Dialogue 1;

[0155] B: (Neutral) Dialogue 2;

[0156] A: (Surprise) Dialogue 3;

[0157] B: (Sad) Dialogue 4;

[0158] B: (Fear, Sad) Dialogue 5;

[0159] A: (Anger) Dialogue 6;

[0160] Figure 6 The six groups of multimodal vectors in correspond to the six sentences of the above dialogue. In the current round, when speaker A is speaking, speaker B does not speak but has facial expressions. Therefore, this situation should be taken into account. For each time step, f a is the multimodal emotional feature subvector of speaker A, and f concatenated with it b The audio and text parts of the vector are both empty, with only the down-weighted subvector of the image modality. The image modality here selects the image of speaker B in the previous round (when the speaker is B, the same process is performed);

[0161] In addition, Figure 6 In the example, speaker A and speaker B can be the same person, and in this case it can be considered that only one speaker is talking to himself.

[0162] The image down-weighted sub-vector corresponding to each speaker is fused with the multimodal emotion feature sub-vectors corresponding to all other speakers to obtain the fused multimodal emotion feature sub-vector corresponding to each speaker;

[0163] The fused multimodal emotion feature sub-vectors corresponding to all speakers are reconnected according to the speaking order of different speakers to obtain the emotion feature vector fused with the temporal features;

[0164] The sentiment feature vector fused with the temporal features is input into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result. The final output sentiment is the sentiment category of the last sentence of each speaker.

[0165] like Figure 6 As shown, the bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier having the same structure and arranged in parallel; the forward RNN classifier and the backward RNN classifier each include a plurality of RNN classification layers connected in sequence; all RNN classification layers have the same structure, and the specific structure is an LSTM network;

[0166] At each time step, the forward RNN processes the input sequence from beginning to end, while the backward RNN processes it from end to beginning. The hidden states in both directions can be concatenated at each time step to form a more comprehensive representation that includes temporal context interaction information.

[0167] Finally, the results of the three sentiment classifications are hard voted according to the rule of minority obeys majority. The text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and temporal context interaction sentiment classification results are hard voted together. The sentiment category with the most votes for each speaker is taken as its final sentiment classification result to complete the multimodal dialogue sentiment recognition. If there are multiple sentiment results with the most occurrences, any one of them can be selected as the final sentiment classification result.

[0168] This method avoids emotional interference by optimizing the multimodal interaction mode, and at the same time models the interaction between historical dialogues and speakers, mining the emotional features contained in each modality in a more detailed way, which can enhance the accuracy and robustness of sentiment classification.

[0169] The same or similar reference numerals correspond to the same or similar components;

[0170] The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present application;

[0171] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multimodal dialogue emotion recognition method based on multi-model voting, characterized in that: The following steps are involved: S1: Acquire multimodal data generated by at least one speaker; the multimodal data includes text data, audio data and image data; each modal data includes at least one emotion category to be identified; S2: Inputting the text data, audio data and picture data into a preset text encoder, audio encoder and picture encoder respectively for feature extraction, and obtaining text features, audio features and picture features respectively; S3: inputting the text features, audio features and picture features into a preset text sentiment classifier, audio sentiment classifier and picture sentiment classifier respectively to perform a first sentiment classification, and obtaining a text sentiment classification result, an audio sentiment classification result and a picture sentiment classification result respectively; S4: Calculate the penalty factor corresponding to each modality according to the text sentiment classification results, the audio sentiment classification results, and the image sentiment classification results; multiply the text features, the audio features, and the image features by the penalty factor corresponding to each modality to obtain the text down-weighting vector, the audio down-weighting vector, and the image down-weighting vector respectively; S5: Input the text down-weighted vector, the audio down-weighted vector, and the image down-weighted vector into the preset multi-head attention layer to perform multi-modal feature fusion interaction to obtain a multi-modal emotion feature vector; S6: input the multimodal emotion feature vector into a preset multimodal emotion classifier for a second emotion classification to obtain a multimodal fusion emotion classification result; S7: decomposing the multimodal emotion feature vector into a plurality of multimodal emotion feature sub-vectors, and rejoining all the multimodal emotion feature sub-vectors in time sequence to obtain an emotion feature vector integrating time sequence features; S8: Input the sentiment feature vector fused with the temporal features into the trained bidirectional RNN classifier for the third sentiment classification to obtain the temporal context interaction sentiment classification result; S9: Hard voting is performed on the text sentiment classification results, audio sentiment classification results, image sentiment classification results, multimodal fusion sentiment classification results and temporal context interaction sentiment classification results. The sentiment category with the largest number of votes for each speaker is taken as the final sentiment classification result to complete multimodal dialogue sentiment recognition.

2. According to the multimodal dialogue emotion recognition method based on multi-model voting according to claim 1, it is characterized in that: The multimodal data in step S1 is obtained by extracting text data, audio data and image data from the conversation video data of at least one preset speaker to obtain the multimodal data.

3. According to claim 2, a multimodal dialogue emotion recognition method based on multi-model voting is characterized in that: The step S1 also includes preprocessing the acquired audio data and image data, and the specific method is as follows: The audio data is sequentially subjected to sampling rate adjustment, denoising and audio gain adjustment, and an audio segment is extracted using a sliding window to complete preprocessing of the audio data; The image data is deduplicated, specifically: S1.1: performing optical flow calculation on two pictures of adjacent frames of the conversation video data using an optical flow algorithm to obtain an optical flow calculation result; S1.2: Calculate the change range of two pictures of adjacent frames according to the optical flow calculation results, and determine whether the change range is greater than a preset threshold. If so, retain the two pictures of the adjacent frames in the deduplication result set and execute step S1.3; Otherwise, directly execute step S1.3; S1.3: Starting from the first frame of the conversation video data, repeat steps S1.1~S1.2 to update the deduplication result set until the last frame of the conversation video data is traversed, and save the deduplication result set obtained from the last update as the deduplication picture data to complete the preprocessing of the picture data.

4. According to claim 3, a multimodal dialogue emotion recognition method based on multi-model voting is characterized in that: The optical flow algorithm in step S1.1 is specifically any one of the Lucas-Kanade algorithm, the Farneback algorithm and the FlowNet algorithm.

5. According to the multimodal dialogue emotion recognition method based on multi-model voting according to claim 4, it is characterized in that: The change amplitude in step S1.2 is specifically the displacement vector size or similarity measure of each pixel in two pictures of adjacent frames; the similarity measure includes Euclidean distance and angle change.

6. The multimodal dialogue emotion recognition method based on multi-model voting according to claim 1 or 5, characterized in that: In the step S2, the preset text encoder is specifically a Transformer encoder, and the Transformer encoder is specifically any one of a BERT model, a MacBERT model, a RoBERTa model, and an ERNIE model; The preset audio encoder is specifically a HuBERT model; The preset picture encoder is specifically a ViT model.

7. The multimodal dialogue emotion recognition method based on multi-model voting according to claim 6 is characterized in that: The text sentiment classifier, audio sentiment classifier and picture sentiment classifier preset in step S3, and the multimodal sentiment classifier preset in step S6 are all CRF classifiers.

8. The multimodal dialogue emotion recognition method based on multi-model voting according to claim 7 is characterized in that: In step S4, the specific method for calculating the penalty factor corresponding to each mode is: According to the text sentiment classification results, audio sentiment classification results and image sentiment classification results, the types of sentiment categories n are obtained, and the total number of occurrences of each sentiment category is counted respectively, and the proportion of the i-th sentiment category is calculated. , specifically: in, is the number of occurrences of the i-th emotion category; For the classification results of each modality, the proportions of all the emotion categories it contains are added together, and the sum is used as the penalty factor of the corresponding modality.

9. The multimodal dialogue emotion recognition method based on multi-model voting according to claim 8 is characterized in that: The specific method of step S7 is: Decomposing the multimodal emotion feature vector into a number of multimodal emotion feature sub-vectors according to different speakers, each multimodal emotion feature sub-vector corresponds to each speaker one by one; The multimodal emotion feature sub-vector includes a text down-weighting sub-vector, an audio down-weighting sub-vector and a picture down-weighting sub-vector; The image down-weighted sub-vector corresponding to each speaker is fused with the multimodal emotion feature sub-vectors corresponding to all other speakers to obtain the fused multimodal emotion feature sub-vector corresponding to each speaker; The fused multimodal emotion feature sub-vectors corresponding to all speakers are reconnected according to the speaking order of different speakers to obtain the emotion feature vector fused with the temporal features.

10. The multimodal dialogue emotion recognition method based on multi-model voting according to claim 9 is characterized in that: The bidirectional RNN classifier in step S8 is specifically: The bidirectional RNN classifier includes a forward RNN classifier and a backward RNN classifier having the same structure and arranged in parallel; The forward RNN classifier and the backward RNN classifier both include a plurality of RNN classification layers connected in sequence; All RNN classification layers have the same structure, which is any one of the LSTM network, GRU network, and PQRNN network.

Images