Video dialogue style identification method based on multi-modal emotion fusion
By deriving and fusing multimodal features and incorporating emotional information, the method improves the recognition of video conversation styles, addressing the limitations of existing methods that focus solely on visual modalities.
Patent Information
- Application Number
- CN202310625569.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-30
AI Technical Summary
Existing video conversation style recognition methods primarily focus on visual modalities, neglecting the importance of audio and text information, and lack effective incorporation of emotions, which are crucial for distinguishing different conversation styles.
A method that derives and fuses multimodal features, including visual, auditory, and text features, and incorporates emotional features using a multi-head attention mechanism to enhance dialogue style recognition, utilizing pre-trained models for emotion extraction and classification.
Enhances the ability to distinguish different conversation styles by leveraging multimodal and emotional features, improving the robustness and accuracy of video dialogue style recognition.
Smart Images

Figure CN119068561A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and relates to the derivation and fusion of multimodal features in videos, emotion analysis, and video classification technology, and specifically is a video dialogue style recognition method based on multimodal emotion fusion. Background Art
[0002] Existing human-centered video analysis focuses on identifying the actions of people, such as action recognition tasks and interaction recognition tasks. However, in addition to physical interaction, conversation is also a very important form of human interaction. Video conversation style recognition refers to the style classification of a video of a conversation between people, such as discussion, teaching, explanation, confrontation, intimidation, etc. It is a fine-grained description of the conversation between people. Conversation style is a complex judgment that requires a comprehensive evaluation of the content of the conversation, the status of the characters in the conversation, the interaction, and other modalities. The recognition of conversation style helps to conduct a deeper study of the conversation between people, so as to build a more comprehensive understanding of the behavior of the characters.
[0003] At present, most of the work on identifying video conversation styles focuses on visual modalities. Some models and identifies the spatial relationship of entities in the video, while others models and identifies the temporal relationship of information in the video. However, the task of video conversation style recognition focuses on character conversations. The volume and content of character conversations have a great impact on the conversation style. Therefore, in the process of video analysis, audio and text information are also particularly important. Extracting appropriate multimodal features and effectively fusing them will provide more powerful help for conversation style recognition.
[0004] Secondly, conversation style can also be regarded as a sub-classification of conversation. Therefore, different conversation styles also have certain similarities. It is necessary to add more discriminative features to assist in identifying conversation styles. Emotions, as one of the starting points of human activities, lead to different interactions between people. In the process of identifying conversation styles, paying corresponding attention to the emotions of the characters will effectively improve the ability to distinguish different conversation styles. Emotion recognition in conversations is becoming a new research frontier. Although some studies have explored the perception of emotional conversations in texts, emotions are not equivalent to conversation styles, but only a factor that affects conversation styles. How to add emotions to the task of video conversation style recognition is still an important challenge. Summary of the invention
[0005] The problem to be solved by the present invention is: how to add emotions to the task of video dialogue style recognition and perform style recognition on the dialogues of characters in the video. Since character dialogues are strongly correlated with visual, auditory and textual information, the present invention derives and fuses multimodal features to analyze dialogue styles; in addition, since different dialogue styles are subcategories of dialogues, they inevitably have a certain degree of similarity, so it is necessary to add more powerful and more discriminating features to improve the effectiveness of multi-dialogue style recognition.
[0006] The technical solution of the present invention is: a video dialogue style recognition method based on multimodal emotion fusion, which recognizes the dialogue style of a video containing character dialogues: first, different feature derivation models are used to derive the visual, auditory and text features of the video, and then a pre-trained multimodal emotion model is used to extract visual emotion features, auditory emotion features and text emotion features, and a multi-head attention mechanism is used to fuse the visual features with the visual emotion features, the auditory features with the auditory emotion features, and the text features with the text emotion features. The visual features with emotions, the auditory features with emotions, and the text features with emotions obtained after the fusion are respectively input into the visual, auditory and text classification networks to obtain visual classification results, auditory classification results and text classification results, and finally these results are fused to obtain the final dialogue style prediction result.
[0007] Further, the present invention comprises the following steps:
[0008] 1) Extract visual features f from the video v ;
[0009] 2) Convert the video into audio and then extract the audio features f a ;
[0010] 3) Extract text from the video, and then extract text features f t ;
[0011] 4) The visual feature f v , auditory features a and text features f t Input the pre-trained multimodal emotion recognition model SelfMM to extract the visual modality emotion feature S v ′ , auditory modality emotion feature S ′ a and text modality sentiment feature S t ′ ;
[0012] 5) The visual modality emotion feature S v ′ Input a convolution and pooling layer to obtain the visual emotion feature S that is more relevant to the conversation style. v; Similarly, the auditory modal emotion feature S ′ a and text modality sentiment feature S t ′ Input the corresponding convolutional pooling layer and follow the same process to obtain the auditory emotion feature S that is more relevant to the conversation style. a and text sentiment feature S t ;
[0013] 6) The visual feature f v Input an LSTM network to model the temporal relationship between video frames, thereby obtaining visual features F with temporal information v ;
[0014] 7) The auditory feature f a Input an LSTM network to model the conversation process between characters, thereby obtaining auditory features F with temporal information a ;
[0015] 8) Tokenize the text t Input a BERT network to obtain the relationship between words in the text, thereby obtaining the text feature F with time sequence information t ;
[0016] 9) The visual feature F with temporal information v and visual emotion features S v Input a multi-head attention network to calculate and fuse the emotional attention to obtain the visual feature X with emotional attention. v , and similarly obtain the auditory feature X with emotional attention a and text features X with sentiment attention t ;
[0017] 10) Visual features with emotional attention X v Input a classifier consisting of two linear hidden layers and one linear output layer to obtain the probabilities of various dialogue styles predicted from a visual perspective, i.e., the classification results from a visual perspective. Similarly, the classification results from an auditory perspective and the classification results from a text perspective are obtained.
[0018] 11) Perform weighted addition of the visual classification results, auditory classification results, and text classification results to obtain the final dialogue style classification result.
[0019] Different from the video emotion recognition in the prior art, the present invention proposes a dialogue style recognition scheme, not just emotion recognition. The present invention introduces emotion as a feature and a means of feature enhancement to improve the perception ability of dialogue style recognition. The present invention includes multimodal feature derivation technology, multimodal feature fusion technology, emotion analysis technology and video classification technology. The multimodal feature derivation technology extracts the visual features, auditory features and text features of the video, corresponding to steps 1)-3) and 6)-8). The emotion analysis technology will derive multimodal emotion features and combine the attention mechanism to bring its influence into other features, corresponding to steps 4)5) and 9). The multimodal feature fusion technology focuses on the different roles of different modalities in the dialogue style recognition task and provides a more suitable feature fusion scheme, corresponding to steps 10)-11). Based on the previous technologies, the video classification technology is designed. After steps 1)-11), the video is classified into dialogue styles.
[0020] The beneficial effects of the present invention are as follows: the present invention provides a solution for identifying the style of video conversations. The conversations of characters in the video are strongly correlated with visual, auditory and textual information. The present invention derives and fuses the features of multiple modalities to identify the conversation style. In addition, due to the similarity of the conversations, the present invention creatively introduces emotional features to help distinguish different conversation styles, further improving the robustness of the method. First, the present invention extracts and fuses the multimodal spatiotemporal features of the video for video recognition. The information of multiple modalities complements each other and is richer than that of single modal information. In addition, the present invention also introduces multimodal emotional features based on the cross-attention mechanism as a guide for attention, so that the final features have emotional relevance, thereby strengthening the features and better identifying the conversation style. Finally, the present invention uses a late fusion method to weightedly fuse the classification results of multiple modalities, effectively addressing the problem of heterogeneous modal features and maximizing the effect of combining multiple modalities. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The architecture of the network is defined and identified for the input and output of the present invention.
[0022] Figure 2 The results of the present invention are compared with those of other methods.
[0023] Figure 3 The present invention compares the results of using different feature combinations and whether or not to introduce emotional features. DETAILED DESCRIPTION
[0024] The present invention proposes a video dialogue style recognition method based on multimodal emotion fusion. First, multimodal features are used to complete the recognition task. Second, emotions are introduced as an attention feature to assist dialogue style recognition. First, different feature extraction models are used to extract visual, auditory and text features from the video. Then, a pre-trained multimodal emotion model is used to extract visual emotion features, auditory emotion features and text emotion features. The multi-head attention mechanism is used to fuse visual features with visual emotion features, auditory features with auditory emotion features and text features with text emotion features. The processed visual features with emotions, auditory features with emotions and text features with emotions are respectively input into the corresponding classification networks to obtain visual classification results, auditory classification results and text classification results. Finally, these results are fused to obtain the final dialogue style prediction results.
[0025] The implementation of the present invention is described in detail below.
[0026] like Figure 1 As shown, the recognition network input of the present invention is the video to be recognized, and the output is the probability of various dialogue style categories. The implementation of the video dialogue style recognition network based on multimodal emotion fusion specifically includes the following steps.
[0027] 1) Input the video into the pre-trained OpenFace model to extract visual features f from the video v ;
[0028] 2) Use ffmpeg command to convert video into audio, and then input the pre-trained librosa model to extract audio features f a ;
[0029] 3) Input the video into the pre-trained deepspeech model to extract text from the video, and then input the text into the pre-trained BERT
[0030] Tokenizer extracts text features f t ;
[0031] 4) The visual feature f v , auditory features a and text features f t Input the pre-trained multimodal emotion recognition model SelfMM to extract visual emotion features, auditory emotion features, and text emotion features:
[0032]
[0033] in Represents the network structure of the SelfMM model, θ SelfMM Represents the pre-trained parameters of the SelfMM model.
[0034] The multimodal emotion recognition model SelfMM uses self-supervised multi-task learning for multimodal sentiment analysis, see the paper "Learning Modality-Specific Representation with Self-Supervised Mulit-Task Learning for Multimodal Sentiment Analysis" (arXiv:2102.04830v1[cs.CL]9Feb2021).
[0035] 5) The visual modality emotion feature S v ′ Input a convolution and pooling layer to obtain the visual emotion feature S that is more relevant to the conversation style. v ; Similarly, the auditory modal emotion feature S ′ a and text modality sentiment feature S t ′ Input the corresponding convolutional pooling layer and follow the same process to obtain the auditory emotion feature S that is more relevant to the conversation style. a and text sentiment feature S t :
[0036]
[0037] in is a convolutional layer, is a maximum pooling layer.
[0038] 6) The visual feature f v Input an LSTM network to model the temporal relationship between video frames, thereby obtaining visual features F with temporal information v ;
[0039] 7) The auditory feature f a Input an LSTM network to model the conversation process between characters, thereby obtaining auditory features F with temporal information a ;
[0040] 8) Tokenize the text t Input a BERT network to obtain the relationship between words in the text, thereby obtaining the text feature F with time sequence information t ;
[0041] 9) The visual feature F with temporal information v and visual emotion features S v Input a multi-head attention network to calculate and fuse the emotional attention to obtain the visual feature X with emotional attention. v, we can also obtain the auditory feature X with emotional attention a and text features X with sentiment attention t , with X v Taking the calculation of as an example, the details are as follows.
[0042] 9.1) The visual feature F with temporal information v Input a linear layer to get the index Q, and convert the visual emotion feature S v Input a linear layer to get the key value K, and convert the visual emotion feature S v Feed this into another linear layer to get the value V:
[0043] Q=F v ×W q +b q
[0044] K=S v ×W k +b k
[0045] V=S v ×W u +b u
[0046] Where W q represents the weight of the linear layer that calculates the index, b q Represents the weight of the linear layer for computing the index, W k represents the weight of the linear layer that calculates the key value, b k Represents the weight of the linear layer that calculates the key value, W u The weights of the linear layer representing the calculated values, b u Represents the weight of the linear layer for computing the value.
[0047] 9.2) Multiply Q and K, divide by a scaling factor, and input into the softmax layer to obtain the feature similarity matrix. Multiply the similarity matrix with V to introduce emotional attention into the visual features:
[0048]
[0049] Where n is the number of attention heads in the multi-head attention network, d is the scaling factor, and T is the matrix transpose operation.
[0050] 9.3) The multi-head emotional attention feature A obtained in the previous step j Concatenate and input a linear layer for update to obtain the complete emotional attention feature:
[0051]
[0052] Where W orepresents the weight of the linear layer used to update the emotional attention feature, b o represents the weight of the linear layer used to update the emotional attention feature, Represents a feature concatenation operation.
[0053] 9.4) Finally, the emotional attention features are added to the visual features and normalized, input into the feedforward network for updating, and then added and normalized again to obtain the final visual features that incorporate emotional attention:
[0054]
[0055]
[0056]
[0057] in represents feature standardization, represents the feedforward network, W1, b1, W2 and b2 are the parameters of the feedforward network.
[0058] Similarly, we obtain the auditory feature X with emotional attention. a and text features X with sentiment attention t .
[0059] 10) Visual features with emotional attention X v By inputting a classifier consisting of two linear hidden layers and one linear output layer, we can obtain the probabilities of various dialogue styles predicted from a visual perspective, that is, the classification results from the visual perspective. Similarly, we can also obtain the classification results from the auditory perspective and the classification results from the text perspective.
[0060] 11) Perform weighted addition of the visual classification results, auditory classification results, and text classification results to obtain the final dialogue style classification result:
[0061] y=k v y v +k a y a +k t y t
[0062] where y v ,y a and t is the classification result of visual, auditory and textual modalities, k v , k a and k t It is the weight of the visual, auditory and text results, representing the influence of different modal features on the final classification result, and is added to the training in the form of a linear layer.
[0063] Obtain a dataset to train the recognition model designed by the above steps 1)-11) to learn to perceive various conversation styles, such as discussion, teaching, explanation, confrontation, intimidation, etc. Finally, the trained recognition model will recognize the conversation style of new videos and identify the conversation style type of the current video based on the predicted classification results.
[0064] The present invention is implemented on the LVU dataset, which includes five types of dialogue styles, namely discussion, confrontation, teaching, threat and explanation. The dataset has a total of 1,339 samples, including 937 training samples, 203 verification samples and 199 test samples. In order to verify the effectiveness of the present invention, we compared the present invention with other methods. In addition, in order to verify the effectiveness of fusing multimodal features and introducing emotional features, we made different combinations of features and compared them. The indicators used are Accuracy and F1-score, which are commonly used indicators for video classification tasks.
[0065] Figure 2 The 1st to 10th rows in the table are respectively: the results of video conversation style recognition using SlowFast, VideoBERT, Mult, VIVIT, MMIM, ObjectTransformer, Swin Transformer, ViS4mer, STAN and MeMViT. The eleventh row MMSF is the result of video conversation style recognition by the present invention. It can be seen that the performance of the present invention in the video conversation style recognition task is much higher than that of other methods, which shows the effectiveness of the video conversation style recognition method based on multimodal emotion fusion proposed by the present invention.
[0066] Figure 3 The V, A, and T in the table header represent visual, auditory, and text features, respectively. Acc and F1 represent the accuracy value and F1-score value, respectively. w / o sentiment and w / sentiment represent the case where multimodal sentiment features are not introduced and the case where multimodal sentiment features are introduced. Figure 3 From the “w / o sentiment” row, we can see that in most cases, using multiple modalities is better than using a single modality when performing video conversation style recognition. In particular, when the three features of vision, hearing, and text are combined, the recognition effect is the best. This shows the effectiveness of multimodal feature fusion for video conversation style recognition.
[0067] from Figure 3Comparison of the “w / o sentiment” row and the “w / sentiment” row shows that, in most cases, the introduction of emotional features has a positive impact on the conversation style recognition task, whether it is a single-modal or multi-modal feature combination. This proves the effectiveness of the present invention incorporating emotional features to assist in conversation style recognition.
Claims
1. A video dialogue style recognition method based on multimodal emotion fusion, characterized by Conversation style recognition is performed on videos containing character conversations: first, different feature derivation models are used to derive the visual, auditory and textual features of the video. Then, the pre-trained multimodal emotion model is used to extract visual emotion features, auditory emotion features and textual emotion features. The multi-head attention mechanism is used to fuse the visual features with the visual emotion features, the auditory features with the auditory emotion features, and the textual features with the textual emotion features. The fused visual features with emotions, auditory features with emotions and textual features with emotions are respectively input into the visual, auditory and textual classification networks to obtain visual classification results, auditory classification results and textual classification results. Finally, these results are fused to obtain the final conversation style prediction result.
2. A video dialogue style recognition method based on multimodal emotion fusion according to claim 1, characterized in that The following steps are involved: 1) Extract visual features f from the video v ; 2) Convert the video into audio and then extract the audio features f a ; 3) Extract text from the video and then extract text features f t ; 4) The visual feature f v , auditory features a and text features f t Input the pre-trained multimodal emotion recognition model SelfMM to extract the visual modality emotion feature S′ v , auditory modality emotion feature S′ a and text modality sentiment feature S′ t ; 5) Visual modality emotion feature S′ v Input a convolution and pooling layer to obtain the visual emotion feature S that is more relevant to the conversation style. v ; Similarly, the auditory modal emotion feature S′ a and text modality sentiment feature S′ t Input the corresponding convolutional pooling layer and follow the same process to obtain the auditory emotion feature S that is more relevant to the conversation style. a and text sentiment feature S t ; 6) The visual feature f v Input an LSTM network to model the temporal relationship between video frames, thereby obtaining visual features F with temporal information v ; 7) The auditory feature f a Input an LSTM network to model the conversation process between characters, thereby obtaining auditory features F with temporal information a ; 8) Tokenize the text t Input a BERT network to obtain the relationship between words in the text, thereby obtaining the text feature F with time sequence information t ; 9) The visual feature F with temporal information v and visual emotion features S v Input a multi-head attention network to calculate and fuse the emotional attention to obtain the visual feature X with emotional attention. v , and similarly obtain the auditory feature X with emotional attention a and text features X with sentiment attention t ; 10) Visual features with emotional attention X v Input a classifier consisting of two linear hidden layers and one linear output layer to obtain the probabilities of various dialogue styles predicted from a visual perspective, that is, the classification results from the visual perspective. Similarly, the classification results from the auditory perspective and the classification results from the text perspective are obtained. 11) Perform weighted addition of the visual classification results, auditory classification results, and text classification results to obtain the final dialogue style classification result.
3. The video dialogue style recognition method based on multimodal emotion fusion according to claim 2 is characterized by: Steps 4) and 5) are specifically as follows: 4) The visual feature f v , auditory features a and text features f t Input the pre-trained multimodal emotion recognition model SelfMM to obtain multimodal emotion features: in Represents the network structure of the SelfMM model, θ SelfMM Represents the pre-trained parameters of the SelfMM model; 5) The multimodal emotion features are then input into the corresponding convolutional pooling layer for updating, so that the captured emotions are more suitable for dialogue style recognition: is a convolutional layer, is a maximum pooling layer.
4. The video dialogue style recognition method based on multimodal emotion fusion according to claim 2 is characterized by: The multi-head attention network described in step 9) is specifically: Visual features with temporal information F v and visual emotion features S v Fusion: 9.1) Input the visual features with temporal information into a linear layer to get the index Q, input the visual emotion features into a linear layer to get the key value K, and input the visual emotion features into another linear layer to get the value V: Q=F v ×W q +b q K=S v ×W k +b k V=S v ×W u +b u Where W q represents the weight of the linear layer that calculates the index, b q Represents the weight of the linear layer for computing the index, W k represents the weight of the linear layer that calculates the key value, b k Represents the weight of the linear layer that calculates the key value, W u The weights of the linear layer representing the calculated values, b u Represents the weight of the linear layer of the calculated value; 9.2) Multiply Q and K, divide by a scaling factor, and input into the softmax layer to obtain the feature similarity matrix. Multiply the similarity matrix with V to introduce emotional attention into the visual features: Where n is the number of attention heads in the multi-head attention network, d is the scaling factor, and T is the matrix transpose operation; 9.3) The obtained multi-head emotional attention feature A j Concatenate and input a linear layer for update to get the complete emotional attention feature: Where W o represents the weight of the linear layer used to update the emotional attention feature, b o represents the weight of the linear layer used to update the emotional attention feature, Represents a feature concatenation operation; 9.4) Add the emotional attention features to the visual features and normalize them, input them into the feedforward network for updating, and then add and normalize them again to obtain the final visual features that incorporate emotional attention: in represents feature standardization, represents the feedforward network, W1, b1, W2 and b2 are the parameters of the feedforward network; Similarly, we obtain the auditory feature X with emotional attention. a and text features X with sentiment attention t .
5. The video dialogue style recognition method based on multimodal emotion fusion according to claim 2 is characterized by: The multi-modal result weighted fusion method described in step 11) is: y=k v and v +k a and a +k t and t where y v ,y a and t is the classification result of visual, auditory and textual modalities, k v , k a and k t It is the weight of the visual, auditory and text results, representing the influence of different modal features on the final classification result, and is added to the training in the form of a linear layer.
6. The video dialogue style recognition method based on multimodal emotion fusion according to claim 2 is characterized in that in step 1), the video is input into a pre-trained OpenFace model to extract visual features f from the video v ; Step 2) Use ffmpeg command to convert the video into audio, and then input the pre-trained librosa model to extract audio features f a ; In step 3), the video is input into the pre-trained deepspeech model to extract text from the video, and then the text is input into the pre-trained BERT Tokenizer to extract text tokens f t .
Citation Information
Patent Citations
Multi-modal sentiment analysis method based on multi-task learning and stacked cross-modal fusion
CN114694076A
Intelligent emotion recognition system based on multiple modes and contexts
CN115827827A
System for assemblying battery cell case
KR102250593B1
Emotion recognition in multimedia videos using multi-modal fusion-based deep neural network
US20230154172A1
