Multi-modal dialogue emotion recognition method based on modal enhancement and interaction of dimension conversion

Abstract

The invention discloses a modal enhancement and interaction multi-modal dialogue emotion recognition method based on dimension conversion, which comprises the following steps of: 1) constructing a modal encoder, constructing a respective modal encoder for the input of each modal, and encoding text data, audio data and video data; 2) constructing a modal enhancement network, extracting interaction information among different modals by the modal enhancement network through a modal enhancement block MABlock connected by a residual network, and capturing characteristics among the modals; meanwhile, different modes are processed respectively, and features in the modes are captured to enhance the representation ability of the modes; 3) constructing a modal interaction network, performing interactive fusion on the intra-modal features and the inter-modal features by the modal interaction network, and providing final fusion features to a classifier; and 4) constructing a classifier, wherein the classifier calculates an option with the maximum probability as a final classification result according to the final fusion feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and particularly relates to a multi-modal dialogue emotion recognition method based on modal enhancement and interaction of dimension conversion. BACKGROUND

[0002] Multi-modal dialogue emotion recognition refers to combining multiple perception modalities (such as text, speech, image, video, etc.) to recognize and understand the emotional and emotional state in the dialogue. In today's era of rapid development of information technology, people more and more use multiple ways to communicate, for example, micro-blog users often publish pictures and text together, TikTok users usually add infectious music to the published video, and Xiaohongshu users add picture descriptions to the picture tweet. These phenomena make it insufficient and inaccurate to rely on only a single modality such as text or speech for emotion recognition. Therefore, multi-modal dialogue emotion recognition emerges as the times require and is widely used in social media analysis, human-computer interaction and other fields, becoming a research direction that attracts much attention.

[0003] However, multi-modal data is often asynchronous, and there is a distribution gap between text, audio and visual modal data, which often introduces information redundancy. In dialogue emotion recognition, most work focuses on context modeling based on sequence models or graph models, while the work of multi-modal fusion is often ignored. Although these early works can achieve multi-modal data fusion through simple early-late fusion, they cannot extract inter-modal interaction information, and simple concatenation operation is easy to introduce redundant information. In recent years, some dialogue emotion recognition works based on deep graph multi-modal fusion have been proposed, but due to the characteristics of deep graph, redundant information is gradually accumulated in each layer of vector space, which is easy to cause over-smoothing problem, resulting in insufficient node differentiation. SUMMARY

[0004] We observed that the distribution of emotional expression in text, audio and visual information has subtle differences, but they often concentrate in a short period of time. For the purpose of discussion, we standardized the embedding dimensions of various modalities, considering the vector length as the time dimension. Therefore, the modality vector can be conceptualized as a one-dimensional time series. Inspired by this, we call the change of adjacent time intervals within a single modality and the change of the same time interval between different modalities as intra-modal change and inter-modal change, respectively. In order to represent these features, we convert the one-dimensional vector to a two-dimensional vector, where the length axis represents intra-modal information and the dimension axis represents inter-modal information. As a result, we successfully merged the intra-modal and inter-modal features into a two-dimensional tensor.

[0005] Based on the above motivation, we propose a multi-modal dialogue emotion recognition model based on modal enhancement and interaction network of dimension conversion, which is used to solve the problems of insufficient inter-modal interaction information and intra-modal information, so that the multi-modal dialogue emotion recognition is more accurate.

[0006] The method for multi-modal dialogue sentiment recognition based on dimension transformation, modal enhancement and interaction network comprises the following steps:

[0007] (5) Constructing a modal encoder:

[0008] A respective modal encoder is constructed for the input of each modal to encode the text data, audio data and video data.

[0009] (6) Constructing a modal enhancement network:

[0010] The modal enhancement network extracts the interaction information between different modalities through the modal enhancement block MABlock connected by a residual network, captures the inter-modal features. At the same time, the different modalities are processed respectively to capture the intra-modal features to enhance the representation ability of the modal itself.

[0011] (7) Constructing a modal interaction network:

[0012] The modal interaction network interacts and fuses the intra-modal features and the inter-modal features to provide the final fusion features to the classifier.

[0013] (8) Constructing a classifier:

[0014] The classifier calculates the option with the maximum probability as the final classification result according to the final fusion features.

[0015] Further, in step (1), each sample data is: an original data containing text data Ut, audio data Ua, video data Uv and classification label Y. The goal of multi-modal sentiment recognition is to predict the emotion classification to which the sample belongs through text, audio and video data.

[0016] Further, in step (1), in order to obtain the encoding of different modal data, for text, the application uses the RoBERTa pre-training model for processing; for audio, the application extracts the mel spectrogram feature through librosa; for video, the application uses the effecientNet pre-trained on the VGGface and AFE data sets to extract the video feature. The extracted three modalities are subjected to one-dimensional convolution to make the dimensions the same, and finally the text feature audio feature video feature wherein T t , T a , T v are the lengths of the text, audio and video features respectively, and d is the feature dimension.

[0017] Further, in step (2), in order to extract the inter-modal features (interaction features) and intra-modal features of the three different modalities, the application uses a multi-modal enhancement network MAN to process the three modalities, and the specific steps are as follows:

[0018] For inter-modal features, for the convenience of discussion, the dimension d of the same modality is ignored and regarded as a one-dimensional feature. First, the three modal features are filled and converted to obtain a two-dimensional feature:

[0019]

[0020] where Padding() is a process of filling the three modalities to the same length, and the padding length is the longest length T δ t a v . Thus, a two-dimensional tensor After conversion, the two-dimensional tensor is subjected to a 2D convolution model Inception with multiple kernel sizes to obtain a global interaction information feature

[0021]

[0022] Then, the learned tensor is converted back to one-dimensional space, and the three tensors with a length of T δ are truncated back to their original lengths:

[0023]

[0024] The application refers to the above process as a modal enhancement block MABlock, which means that the interaction information between different modalities is enhanced. The modal enhancement block is connected by a residual connection to form a modal enhancement network MAN, so as to prevent the gradient vanishing problem in the deep network and improve the network expression ability. Specifically, given three tensors with lengths T t , T a , T v , they are represented by one-dimensional tensors with a length of T δ , and for the k-th layer MAN, the input is The calculation process is as follows:

[0025]

[0026] The overall calculation process of the k-th layer MAN can be summarized as follows:

[0027]

[0028] ​​​

[0029]

[0030] After k layers of MABlock processing, three modal outputs are obtained Finally, the three outputs are averaged to obtain the final output: In addition, we input the text, audio and video features into MAN respectively, that is, we use three MANs to process the three modalities respectively, denoted as

[0031] Further, in step (3), the application proposes cross-modal attention, the main idea of which is to reconstruct modality a using modality β to realize the fusion of the two modalities. Specifically, modality β tensor X β As the input of K and V in attention, modality a is the input of Q, and cross-modal attention Y α is defined as follows:

[0032]

[0033] The cross-modal attention is stacked to form a cross-modal Transformer (CMT):

[0034]

[0035] In addition, in order to fuse the inter-modal and intra-modal information, the application also performs a traditional self-attention-based Transformer (SMT) processing on each modality. Finally, the two are averaged to obtain the comprehensive representation of each modality:

[0036]

[0037] wherein is the inter-modal feature X c and the intra-modal feature of modality a is the output obtained by CMT, is the output obtained by SMT, wherein α∈{t, a, v}.

[0038] Further, in step (4), first, Y t , Y a , Y v are spliced according to the length dimension, and then two fully connected layers and softmax are performed for classification, wherein the activation function is GELU:

[0039]

[0040]

[0041] Wherein W1, W2, b1, b2 are learnable parameters. The loss function of the application selects cross-entropy loss:

[0042]

[0043] Wherein is the predicted result, y i is the real result, N b is the batch size, that is, the number of samples in a batch.

[0044] Compared with the prior art, the application has the beneficial effects that:

[0045] (1) In view of the insufficient inter-modal information capturing ability, the modal enhancement network of step (2) is used to effectively fuse the three modalities, inter-modal features are extracted by means of splicing-convolution-pooling-splitting of the three modalities, and the problem of gradient disappearance is avoided by using residual link.

[0046] (2) In view of the insufficient intra-modal information capturing ability, similarly, the modal enhancement network of step (2) is used to enhance the three modalities respectively, and intra-modal features are extracted by means of self-modal replication-splicing-convolution-pooling-splitting.

[0047] (3) In view of the effective fusion of inter-modal features and intra-modal features, the modal interaction network of step (3) is used to interact and fuse the inter-modal features and the intra-modal features by means of cross-modal Transformer and self-modal Transformer, so as to generate rich features with global information and local information. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the abstract drawing

[0049] Figure 2 is the overall architecture diagram of the application

[0050] Figure 3 is the modal enhancement network MAN network architecture diagram of the application

[0051] Figure 4 is the cross-modal Transformer architecture diagram of the application DETAILED DESCRIPTION

[0052] The application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0053] The implementation method of the application is given taking the multi-modal sentiment dialogue data set MELD as an example. The overall framework of the method is shown in Figure 2The whole system algorithm flow includes five steps of MELD dataset processing, multi-modal data (text, audio, video) encoding, modal enhancement, modal interaction fusion and multi-modal emotion classification.

[0054] The specific steps are as follows:

[0055] (1) MELD dataset processing:

[0056] MELD contains 13708 utterances in 1433 dialogues of the TV series Friends, each sentence contains an emotion label, and contains text, audio and visual modalities. The dataset contains seven emotions: Anger, Disgust, Fear, Happy, Neutral, Sadness and Surprise. In order to make a fair comparison, the experiments we conducted used the pre-set training set, validation set and test set division of the MELD dataset.

[0057] (2) Multi-modal data encoding

[0058] The present application defines each sample data as: an original data containing text data U t , audio data U a , video data U v and classification label Y. The goal of multi-modal emotion recognition is to predict the emotion classification to which the sample belongs through text, audio and video data. In order to obtain the encoding of different modal data, for text, the present application uses the RoBERTa pre-training model for processing; for audio, the present application extracts the Mel spectrogram feature through librosa; for video, the present application uses the efficientNet pre-trained on VGGface and AFE dataset to extract video features. The extracted three modalities are made to have the same dimension through one-dimensional convolution, and finally the text feature audio feature video feature wherein T t , T a , T v are the lengths of the text, audio and video features respectively, and d is the feature dimension.

[0059] (3) Modal enhancement

[0060] For the same modality, each time point involves both the adjacent area within the modality and the same phase change between different modalities, i.e. the change within the modality and the change between modalities. In order to capture both features, the present application constructs a module modal enhancement block (MABlock), which explores the rich information between different modalities through dimension change.

[0061] For inter-modal feature, for the convenience of discussion, the dimension d of the same modality is ignored and regarded as one-dimensional feature. Firstly, the three modal features are filled and converted to obtain a two-dimensional feature:

[0062]

[0063] where Padding() is a process of filling three modalities to the same length, and the filling length is the longest length T δ t a v . Thus a two-dimensional tensor After conversion, the two-dimensional tensor is subjected to a 2D convolution model Inception with multiple kernel sizes to obtain a global interaction information feature

[0064]

[0065] Then the learned tensor is converted back to one-dimensional space, and the three tensors with length T δ are truncated back to their original length:

[0066]

[0067] The above process is referred to as a modal enhancement block MABlock, which means that the interaction information between different modalities is enhanced, and the specific details are shown in Figure 3 . The modal enhancement block is connected by residual connection to form a modal enhancement network MAN to prevent the gradient vanishing problem in the deep network and improve the network expression ability. Specifically, given three t , T a , T v They are represented by one-dimensional tensors δ with length T β , and for the k-th layer MAN, the input is The calculation process is as follows:

[0068]

[0069] The overall calculation process of the k-th layer MAN can be summarized as follows:

[0070]

[0071]

[0072]

[0073] ​​​​After k layers of MABlock processing, three modal outputs are obtained Finally, the three outputs are averaged: In addition, we input the text, audio and video features into MAN respectively, that is, we use three MANs to process the three modalities respectively, denoted as

[0074] In summary, the inter-modal feature calculation method is as follows:

[0075]

[0076] The intra-modal feature calculation method of text, audio and video is as follows:

[0077]

[0078]

[0079]

[0080] (4) Modal interaction fusion

[0081] In order to make full use of the consistent and complementary information in the obtained inter-modal and intra-modal features, the present application proposes a modal interaction network composed of cross-modal Transformer and self-modal Transformer, and the intra-modal features guide the strengthening of inter-modal features.

[0082] The main idea of cross-modal attention is to reconstruct modality β from modality α to realize the fusion of the two modalities. Specifically, as shown in Figure 4 The modality ρ tensor X β As the input of K and V in attention, modality α is the input of Q, and cross-modal attention Y α is defined as follows:

[0083]

[0084] Stacking the cross-modal attention constitutes the cross-modal Transformer (CMT):

[0085]

[0086] In addition, in order to fuse inter-modal and intra-modal information, the present application also performs traditional self-attention-based Transformer (SMT) processing on each modality. Finally, the two are averaged to obtain the comprehensive representation of each modality:

[0087]

[0088] Wherein is the inter-modal feature X c and intra-modal features of modality a obtained by CMT, is the output obtained by SMT of modality a, where a e {t, a, v}.

[0089] (5) Multi-modal sentiment classification

[0090] The sentiment classifier uses three modal features in series as input to perform sentiment prediction. Specifically, Y t , Y a , Y v are spliced according to the length dimension, and then mapped through a fully connected layer to enhance the representation ability, and the activation function is GELU; then after passing through a fully connected layer again, it is classified through a softmax layer:

[0091]

[0092]

[0093] where W1, W2, b1, b2 are learnable parameters. The loss function of the present application selects cross-entropy loss:

[0094]

[0095] where is the predicted result, y i is the true result, and N b is the batch size, i.e. the number of samples in a batch.

[0096] In the specific implementation process, the size of the RoBERTa hidden vector is 768, the dimension of all original text embedding is set to 768, the dimension of all original audio embedding is set to 64, the dimension of all original video embedding is set to 64, the one-dimensional convolution embedding dimension is 30, the cross-modal Transformer layer number is 5, the learning rate is 1e-4, the batchsize is 96, and the optimizer is Adam. The training of the model is carried out on Nvidia GeForce RTX 3090 GPU. The whole training time is about 3 hours.

[0097] Table 1 shows the results of the present model (AIMDiT) and other models (TFN, LMF, DialogueCRN, DialogueGCN, MMGCN, MM-DFN) on the MELD dataset in two evaluation indexes (Acc-7, w-F1). The best performance is indicated in bold.

[0098] Table 1 Experimental results of MELD dataset

[0099]

[0100] The comparative experimental results in the table are described as follows:

[0101] TFN uses an early fusion method for modal fusion. LMF uses low-rank arrangement to fuse tensors. DialogueGCN uses graph structure context dependency to model conversation data. DialogueCRN designs a multi-turn reasoning module to understand the dialogue context. MMGCN uses the structural features of the graph to capture intra-modal and inter-modal features. MM-DFN is a graph-based dynamic fusion module for multi-modal context fusion in dialogue, and reduces redundant information by capturing context dynamics.

[0102] It can be seen that the accuracy and F1 score of AIMDiT on the MELD dataset are better than those of previous methods. Compared with previous models, AIMDiT improves the Acc-7 and w-F1 of MELD by 2.35% and 2.87%, respectively. In order to show more detailed scores, the present application also provides w-F1 scores for each category except the Disgust and Fear categories (which have fewer samples and lack reference). As can be seen from the results, the model proposed by the present application is superior to many complex fusion mechanisms, such as TFN and LFN, and a series of graph-based fusion models, which illustrates the importance of multi-modal fusion.

[0103] In order to illustrate the contribution ability of different modalities to the model, the present application conducts an ablation experiment for different modality combinations, as shown in Table 2. It can be seen that the two-peak and three-peak models are generally superior to the single-peak model. In the two-peak model, the combination of text and audio shows the best performance, indicating that text and audio features are more complementary. In contrast, audio-visual effects are the worst, reflecting the high distribution difference and redundancy of audio-visual features, which also deserves future improvement.

[0104] The MAN module has the ability to extract inter-modal and intra-modal features, and the MIN has the ability to fuse the two features, in order to explore the importance of these two modules, we conduct an ablation experiment on these two modules, and the results are shown in Table 3. The ablation experimental results in the table are described as follows:

[0105] -w / o MAN: a simplified version of removing the modal enhancement network MAN.

[0106] -w / o MIN: a simplified version of removing the modal interaction network MIN.

[0107] It can be seen that after removing MAN and MIN, the performance of both is significantly lower than that of the complete model, especially after removing MAN, the decrease is more than 3%, reflecting the ability of MAN to extract features.

[0108] Table 2 Results comparison of different modalities combination according to Acc-7 and w-F1 metrics

[0109]

[0110] Table 3 Ablation experiments for MAN and MIN

[0111]

[0112] Although the present application has been described with reference to the accompanying drawings, the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and many modifications can be made by those skilled in the art without departing from the spirit of the present application, and these all belong to the protection of the present application.

Claims

1. A multimodal dialogue emotion recognition method based on modality enhancement and interaction of dimensionality conversion, characterized by: The following steps are involved: (1) Constructing a modal encoder: A respective modality encoder is constructed for the input of each modality to encode text data, audio data, and video data. (2) Constructing a modal enhancement network: The modality enhancement network extracts the interaction information between different modalities through modality enhancement blocks (MABlocks) connected by a residual network, capturing inter-modal features. It also processes each modality individually, capturing intra-modal features to enhance the representation capabilities of the modality itself. (3) Constructing a modal interaction network: The modal interaction network interactively fuses intra-modal features and inter-modal features, and provides the final fused features to the classifier. (4) Construct a classifier: The classifier calculates the option with the highest probability based on the final fusion features as the final classification result.

2. The multimodal dialogue emotion recognition method based on dimensional conversion modality enhancement and interaction according to claim 1 is characterized in that: In step (1), each sample data is: an original data containing text data U t , audio data U a , video data U v , classification label Y. The goal of multimodal emotion recognition is to comprehensively predict the emotion category to which the sample belongs through text, audio and video data.

3. The multimodal dialogue emotion recognition method based on dimensional conversion modality enhancement and interaction according to claim 1 or 2 is characterized in that: In step (1), in order to obtain the encoding of different modal data, for text, the present invention uses the RoBERTa pre-trained model for processing; for audio, the present invention uses librosa to extract the Mel spectrogram features; for video, the present invention uses efficientNet pre-trained on the VGGface and AFE datasets to extract video features. The three extracted modalities are subjected to one-dimensional convolution to make them of the same dimension, and finally the text features are obtained Audio characteristics Video Features Where T t , T a , T v are the lengths of text, audio, and video features respectively, and d is the feature dimension.

4. The multimodal dialogue emotion recognition method based on dimensional conversion modality enhancement and interaction according to claim 1 is characterized in that: In step (2), in order to extract the inter-modal features (interaction features) and intra-modal features of the three different modalities, the present invention uses multiple modality enhancement networks MAN to process the three modalities. The specific steps are as follows: For inter-modal features, for the sake of convenience, the dimension d of the same modality is ignored and regarded as a one-dimensional feature. First, the three modal features are filled and transformed to obtain a two-dimensional feature: Padding() is the process of filling the three modes to the same length, and the filling length is the longest length T of the three. δ =max(T t , T a , T v ). This results in a two-dimensional tensor After the conversion, the two-dimensional tensor is passed through a 2D convolution model Inception with multiple core sizes to obtain a global interactive information feature. You will learn later The tensor is converted back to one-dimensional space and the length is T δ The three tensors are truncated back to their original length: The present invention calls the above process a modality enhancement block MABlock, which means that the interaction information between different modalities is enhanced. The modality enhancement blocks are connected with residuals to form a modality enhancement network MAN to prevent the gradient vanishing problem in deep networks and improve the network expression ability. Specifically, given three lengths T t , T a , T v of They are composed of a length of T δ A one-dimensional tensor of Represented, and for the k-th layer MAN, the input is The calculation process is as follows: The overall calculation process of MAN at the kth layer can be summarized as follows: After k layers of MABlock processing, three modal outputs are obtained Finally, the three outputs are averaged and fused: In addition, we also use text, audio and video feature inputs as the input of MAN, that is, use three MANs to process the three modalities respectively, which are expressed as 5. The multimodal dialogue emotion recognition method based on dimensional conversion modality enhancement and interaction according to claim 1 is characterized in that: In step (3), the present invention proposes cross-modal attention, the main idea of ​​which is to use modality β to reconstruct modality α to achieve the fusion of the two modalities. Specifically, the modality β tensor X β As the input of K and V in the attention, the modality α as the input of Q, and the cross-modal attention Y α The definition is as follows: Stacking cross-modal attention constitutes the Cross-Modal Transformer (CMT): In addition, to fuse inter-modal and intra-modal information, the present invention also performs traditional self-attention-based Transformer (SMT) processing on each modality. Finally, the two are averaged to obtain a comprehensive representation of each modality: in is the inter-modal feature X c The output of the intra-modal features of modality α obtained through CMT, is the output of the modality α obtained by SMT, where α∈{t,a,v}.

6. The multimodal dialogue emotion recognition method based on dimensional conversion modality enhancement and interaction according to claim 1 is characterized in that: In step (4), first, Y t , Y a , Y v The data is concatenated according to the length dimension and then classified through two fully connected layers and softmax, where the activation function is GELU: Where W1, W2, b1, b2 are learnable parameters. The loss function of the present invention uses cross entropy loss: in is the prediction result, y i is the true result, N b is the batchsize, that is, the number of samples in a batch.

Images