A video dialogue style recognition method based on multi-modal emotion fusion

Abstract

A video dialogue style recognition method based on multi-modal emotion fusion is used to predict and identify the dialogue style of characters in a movie clip: different feature extraction models are used to extract visual, auditory and text features from the video, and a pre-trained multi-modal emotion model is used to extract visual emotion features, auditory emotion features and text emotion features; a multi-head attention mechanism is used to fuse visual features with visual emotion features, auditory features with auditory emotion features, and text features with text emotion features; the processed emotional visual features, emotional auditory features and emotional text features are input into the corresponding classification network to obtain visual classification results, auditory classification results and text classification results; finally, these results are fused to obtain the final dialogue style prediction result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to the extraction and fusion of multimodal features in videos, emotion analysis, and video classification technology. Specifically, it is a video dialogue style recognition method based on multimodal emotion fusion. Background Technology

[0002] Existing human-centered video analytics focuses on recognizing human actions, such as motion recognition tasks and interaction recognition tasks. However, besides physical interaction, dialogue is also a crucial form of human interaction. Video dialogue style recognition refers to classifying the style of a video containing human dialogue, such as discussion, instruction, explanation, confrontation, or intimidation. It's a fine-grained description of dialogue. Dialogue style is a complex judgment that requires a comprehensive evaluation of multiple modalities, including the content of the dialogue, the state of the characters, and their interactions. Dialogue style recognition facilitates deeper research into human dialogue, leading to a more comprehensive understanding of human behavior.

[0003] Current work on video dialogue style recognition largely focuses on visual modalities, with some modeling spatial relationships between entities and others modeling temporal relationships. However, video dialogue style recognition primarily targets dialogue between individuals, and the volume and content of these dialogues significantly influence dialogue style. Therefore, audio and textual information are also crucial in video analysis. Extracting appropriate multimodal features and effectively fusing them will provide more powerful assistance to dialogue style recognition.

[0004] Secondly, dialogue style can also be seen as a sub-category of dialogue. Therefore, different dialogue styles share certain similarities, requiring the addition of more discriminative features to aid in style identification. Emotion, as one of the starting points of human activity, leads to different interactions between people. Paying attention to the emotions of characters during dialogue style identification will effectively improve the ability to distinguish different dialogue styles. Emotion recognition in dialogue is becoming a new frontier of research. Although some studies have explored the perception of emotion in textual dialogue, emotion is not equivalent to dialogue style, but only one factor influencing it. How to incorporate emotion into video dialogue style recognition tasks remains a significant challenge. Summary of the Invention

[0005] The problem this invention aims to solve is: how to incorporate emotion into video dialogue style recognition tasks to perform style identification on dialogues between characters in videos. Since dialogue is strongly correlated with visual, auditory, and textual information, this invention derives and integrates multimodal features to analyze dialogue styles. Furthermore, because different dialogue styles belong to subcategories of dialogue, they inevitably exhibit a certain degree of similarity, necessitating the addition of more powerful and discriminative features to improve the effectiveness of multi-dialogue style recognition.

[0006] The technical solution of this invention is as follows: a video dialogue style recognition method based on multimodal emotion fusion, which performs dialogue style recognition on videos containing dialogue between characters: First, visual, auditory, and textual features of the video are derived using different feature derivation models. Then, visual emotion features, auditory emotion features, and textual emotion features are extracted using a pre-trained multimodal emotion model. Visual features are fused with visual emotion features, auditory features are fused with auditory emotion features, and textual features are fused with textual emotion features using a multi-head attention mechanism. The resulting emotionally charged visual features, emotionally charged auditory features, and emotionally charged textual features are then input into visual, auditory, and textual classification networks, respectively, to obtain visual classification results, auditory classification results, and textual classification results. Finally, these results are fused to obtain the final dialogue style prediction result.

[0007] Furthermore, the present invention includes the following steps:

[0008] 1) Extract visual features f from the video v ;

[0009] 2) Convert the video to audio, and then extract the audio features f. a ;

[0010] 3) Extract text from the video, then extract text features using r. t ;

[0011] 4) Visual features f v Auditory characteristics f a and text features f t Inputting a pre-trained multimodal emotion recognition model, SelfMM, extracts visual modal emotion features S. v ′ Auditory modality emotional characteristics S ′ a and text modal sentiment features S t ′ ;

[0012] 5) Visual modality emotion features S v ′ Input a convolutional and pooling layer to obtain visual emotion features S that are more relevant to the dialogue style. vSimilarly, auditory modality emotional features S ′ a and text modal sentiment features S t ′ Input the corresponding convolutional pooling layer, and obtain the auditory emotion features S that are more relevant to the dialogue style by following the same process. a and text sentiment features S t ;

[0013] 6) Visual features f v Input an LSTM network to model the temporal relationships between video frames, thereby obtaining visual features F with temporal information. v ;

[0014] 7) Auditory features f a Input an LSTM network to model the dialogue between characters, thereby obtaining auditory features F with temporal information. a ;

[0015] 8) Transfer the text tokens f t Input a BERT network to obtain the relationships between words in the text, thereby obtaining text features F with temporal information. t ;

[0016] 9) Visual features F with temporal information v and visual emotional features S v Input a multi-head attention network, compute and fuse emotional attention to obtain visual features X with emotional attention. v Similarly, auditory features X with emotional attention can be obtained. a And text features X with emotional attention t ;

[0017] 10) Visual features X that attract emotional attention v Input a classifier consisting of two linear hidden layers and one linear output layer to obtain the probability of each dialogue style predicted from the visual perspective, i.e., the classification result from the visual perspective. Similarly, obtain the classification results from the auditory perspective and the classification results from the text perspective.

[0018] 11) The visual classification results, auditory classification results, and text classification results are weighted and summed to obtain the final dialogue style classification result.

[0019] Unlike existing video emotion recognition technologies, this invention proposes a dialogue style recognition scheme, rather than simply identifying emotions. This invention introduces emotion as a feature, using it as a means of feature enhancement to improve the perceptual ability of dialogue style recognition. This invention includes multimodal feature extraction technology, multimodal feature fusion technology, emotion analysis technology, and video classification technology. Multimodal feature extraction technology extracts visual, auditory, and textual features from the video, corresponding to steps 1)-3) and 6)-8). Emotion analysis technology derives multimodal emotion features and incorporates their influence into other features using an attention mechanism, corresponding to steps 4)-5) and 9). Multimodal feature fusion technology focuses on the different roles of different modalities in the dialogue style recognition task, providing a more suitable feature fusion scheme, corresponding to steps 10)-11). Based on the preceding technologies, a video classification technology is designed, and through steps 1)-11), the video is classified according to dialogue style.

[0020] The beneficial effects of this invention are as follows: This invention provides a scheme for identifying video dialogue styles. The dialogue in a video is strongly correlated with visual, auditory, and textual information. This invention derives and fuses features from multiple modalities to identify dialogue styles. Furthermore, due to the similarity of dialogues, this invention creatively introduces emotional features to help distinguish different dialogue styles, further improving the robustness of the method. First, this invention extracts and fuses multimodal spatiotemporal features from the video for video recognition. Multiple modalities complement each other, resulting in richer information compared to single-modal methods. In addition, this invention introduces multimodal emotional features based on a cross-attention mechanism, acting as an attention guide to imbue the final features with emotional relevance, thereby strengthening the features and improving the effectiveness of dialogue style identification. Finally, this invention uses a late-stage fusion method to weightedly fuse the classification results of multiple modalities, effectively addressing the problem of heterogeneous modal features and maximizing the combined effect of multiple modalities. Attached Figure Description

[0021] Figure 1 This describes the architecture of the input / output definition and recognition network for this invention.

[0022] Figure 2 The results of this invention are compared with those of other methods.

[0023] Figure 3 This invention compares the results of using different combinations of features and whether or not emotional features are introduced. Detailed Implementation

[0024] This invention proposes a video dialogue style recognition method based on multimodal emotion fusion. First, it utilizes multimodal features to complete the recognition task; second, it introduces emotion as an attention feature to assist dialogue style recognition. Firstly, different feature extraction models are used to extract visual, auditory, and textual features from the video. Then, a pre-trained multimodal emotion model is used to extract visual, auditory, and textual emotion features. A multi-head attention mechanism is used to fuse visual features with visual emotion features, auditory features with auditory emotion features, and textual features with textual emotion features. The processed emotion-infused visual, auditory, and textual features are then input into their respective classification networks to obtain visual, auditory, and textual classification results. Finally, these results are fused to obtain the final dialogue style prediction result.

[0025] The implementation of this invention is described in detail below.

[0026] like Figure 1 As shown, the present invention identifies the video dialogue style recognition network as input and outputs the probabilities of various dialogue style categories. The implementation of the video dialogue style recognition network based on multimodal emotion fusion specifically includes the following steps.

[0027] 1) Input the video into a pre-trained OpenFace model to extract visual features f from the video. v ;

[0028] 2) Use ffmpeg commands to convert the video to audio, then input it into a pre-trained librosa model to extract audio features f. a ;

[0029] 3) Input the video into the pre-trained DeepSpeech model to extract text from the video, and then input the text into the pre-trained BERT model.

[0030] Tokenizer extracts text features f t ;

[0031] 4) Visual features f v Auditory characteristics f a and text features f t The pre-trained multimodal emotion recognition model SelfMM extracts visual emotion features, auditory emotion features, and textual emotion features:

[0032]

[0033] in Represents the network structure of the SelfMM model, θ SelfMM These represent the pre-trained parameters of the SelfMM model.

[0034] The multimodal emotion recognition model SelfMM applies self-supervised multi-task learning to multimodal sentiment analysis, as described in the paper "Learning Modality-Specific Representation with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis" (arXiv:2102.04830v1[cs.CL]9Feb2021).

[0035] 5) Visual modality emotion features S v ′ Input a convolutional and pooling layer to obtain visual emotion features S that are more relevant to the dialogue style. v Similarly, auditory modality emotional features S ′ a and text modal sentiment features S t ′ Input the corresponding convolutional pooling layer, and obtain the auditory emotion features S that are more relevant to the dialogue style by following the same process. a and text sentiment features S t :

[0036]

[0037] in It is a convolutional layer. It is a max pooling layer.

[0038] 6) Visual features f v Input an LSTM network to model the temporal relationships between video frames, thereby obtaining visual features F with temporal information. v ;

[0039] 7) Auditory features f a Input an LSTM network to model the dialogue between characters, thereby obtaining auditory features F with temporal information. a ;

[0040] 8) Transfer the text tokens f t Input a BERT network to obtain the relationships between words in the text, thereby obtaining text features F with temporal information. t ;

[0041] 9) Visual features F with temporal information v and visual emotional features S v Input a multi-head attention network, compute and fuse emotional attention to obtain visual features X with emotional attention. vSimilarly, auditory features X with emotional attention can be obtained. a And text features X with emotional attention t , with X v Taking the calculation as an example, the specific details are as follows.

[0042] 9.1) Visual features F with temporal information v Input a linear layer to obtain the index Q, and then input the visual emotion features S. v Input a linear layer to obtain key values ​​K, and then input the visual emotion features S. v Inputting another linear layer yields the value V:

[0043] Q = F v ×W q +b q

[0044] K = S v ×W k +b k

[0045] V = S v ×W u +b u

[0046] Among them W q b represents the weight of the linear layer of the computed index. q W represents the weighting of the linear layer of the computed index. k b represents the weights of the linear layer that computes the key values. k W represents the bias of the linear layer that computes key values. u b represents the weights of the linear layer representing the calculated values. u The weighting of linear layers representing the calculated values.

[0047] 9.2) Multiply Q and K, then divide by a scaling factor, and input the result into a softmax layer to obtain a feature similarity matrix. Multiply the similarity matrix by V to incorporate emotional attention into the visual features.

[0048]

[0049] Where n is the number of attention heads in the multi-head attention network, d is the scaling factor, and T is the matrix transpose operation.

[0050] 9.3) The multi-headed emotion attention feature A obtained in the previous step j The features are concatenated and then fed into a linear layer for updating, resulting in complete emotion and attention features:

[0051]

[0052] Among them W ob represents the weights of the linear layer used to update the emotion attention features. o This represents the bias of the linear layer used to update emotional attention features. Representative feature splicing operation.

[0053] 9.4) Finally, the emotional attention features and visual features are added together and standardized, then fed into the feedforward network for updating. The sums are then added together and standardized again to obtain the final visual features incorporating emotional attention.

[0054]

[0055]

[0056]

[0057] in Representative feature standardization, This represents a feedforward network, where W1, b1, W2, and b2 are the parameters of the feedforward network.

[0058] Similarly, auditory features X with emotional attention can be obtained. a And text features X with emotional attention t .

[0059] 10) Visual features X that attract emotional attention v By inputting a classifier consisting of two linear hidden layers and one linear output layer, we can obtain the probabilities of various dialogue styles predicted from a visual perspective, i.e., the classification results from a visual perspective. Similarly, we can obtain the classification results from an auditory perspective and the classification results from a textual perspective.

[0060] 11) The visual classification results, auditory classification results, and text classification results are weighted and summed to obtain the final dialogue style classification result:

[0061] y = k v y v +k a y a +k t y t

[0062] Where y v y a and y t The classification results are for visual, auditory, and textual modalities, k v k a and k t These are the weights of the visual, auditory, and textual results, representing the influence of different modal features on the final classification result, and are added to the training layer in the form of a linear layer.

[0063] The dataset is used to train the recognition model designed in steps 1)-11) above. The model learns to perceive various dialogue styles, such as discussion, teaching, explanation, confrontation, and intimidation. Finally, the trained recognition model performs dialogue style recognition on new videos and identifies the dialogue style type of the current video based on the predicted classification results.

[0064] This invention is implemented on the LVU dataset, which includes five dialogue style categories: discussion, confrontation, instruction, threat, and explanation. The dataset contains 1339 samples: 937 training samples, 203 validation samples, and 199 test samples. To verify the effectiveness of this invention, we compared it with other methods. Furthermore, to verify the effectiveness of fusing multimodal features and incorporating emotion features, we compared different combinations of features. The metrics used are Accuracy and F1-score, commonly used in video classification tasks.

[0065] Figure 2 Rows 1-10 of the table represent the results of video dialogue style recognition using SlowFast, VideoBERT, Mult, VIVIT, MMIM, ObjectTransformer, Swin Transformer, ViS4mer, STAN, and MeMViT, respectively. Row 11, MMSF, represents the results of video dialogue style recognition using this invention. It can be seen that the performance of this invention on the video dialogue style recognition task is significantly higher than other methods, demonstrating the effectiveness of the proposed multimodal emotion fusion-based video dialogue style recognition method.

[0066] Figure 3 In the table header, V, A, and T represent visual, auditory, and textual features, respectively; Acc and F1 represent the Accuracy and F1-score values, respectively; and w / o sentiment and w / sentiment represent the use of multimodal sentiment features and the use of multimodal sentiment features, respectively. Figure 3 The "w / o sentiment" line shows that in video dialogue style recognition, using multiple modalities generally yields better results than using a single modality, especially when visual, auditory, and textual features are combined, resulting in the best recognition performance. This demonstrates the effectiveness of multimodal feature fusion for video dialogue style recognition.

[0067] from Figure 3A comparison of the "w / o sentiment" and "w / sentiment" rows shows that, in most cases, whether it is a combination of unimodal or multimodal features, the introduction of emotion features has a positive impact on the dialogue style recognition task. This proves the effectiveness of the present invention in incorporating emotion features to help with dialogue style recognition.

Claims

1. A video dialogue style recognition method based on multimodal emotion fusion, characterized by: Dialogue style recognition is performed on videos containing dialogue. First, visual, auditory, and textual features of the video are derived using different feature derivation models. Then, a pre-trained multimodal emotion model is used to extract visual, auditory, and textual emotion features. A multi-head attention mechanism is used to fuse visual features with visual emotion features, auditory features with auditory emotion features, and textual features with textual emotion features. The resulting emotionally charged visual, auditory, and textual features are then input into visual, auditory, and textual classification networks, respectively, to obtain visual, auditory, and textual classification results. Finally, these results are fused to obtain the final dialogue style prediction result. The process includes the following steps: 1) Extracting visual features from the video ; 2) Convert the video to audio, and then extract the audio features. ; 3) Extract text from the video, and then extract text features. ; 4) Visual features Auditory characteristics and text features The SelfMM pre-trained multimodal emotion recognition model extracts visual modal emotion features. Auditory modality emotional characteristics and text modal sentiment features ; 5) Visual modality emotional features Input a convolutional and pooling layer to obtain visual emotion features that are more relevant to the dialogue style. Similarly, auditory modality emotional features and text modal sentiment features Input the corresponding convolutional pooling layer and follow the same process to obtain auditory emotion features that are more relevant to the dialogue style. and text sentiment features ; 6) Visual features By inputting an LSTM network, the temporal relationships between video frames are modeled to obtain visual features with temporal information. ; 7) Auditory characteristics Input an LSTM network to model the dialogue between characters, thereby obtaining auditory features with temporal information. ; 8) Transfer text tokens Input a BERT network to obtain the relationships between words in the text, thereby obtaining text features with temporal information. ; 9) Visual features with temporal information and visual emotional characteristics Input a multi-head attention network, compute and fuse emotional attention, thereby obtaining visual features with emotional attention. Similarly, auditory features with emotional attention can be obtained. and text features with emotional attention ; 10) Visual features that attract emotional attention Input a classifier consisting of two linear hidden layers and one linear output layer to obtain the probability of each dialogue style predicted from the visual perspective, i.e., the classification result from the visual perspective. Similarly, obtain the classification results from the auditory perspective and the classification results from the text perspective. 11) The visual classification results, auditory classification results, and text classification results are weighted and summed to obtain the final dialogue style classification result.

2. The video dialogue style recognition method based on multimodal emotion fusion according to claim 1, characterized in that: Steps 4) and 5) are as follows: 4) Visual features Auditory characteristics and text features Input the pre-trained multimodal emotion recognition model SelfMM to obtain multimodal emotion features: in The network structure representing the SelfMM model, Represents the pre-trained parameters of the SelfMM model; 5) The multimodal emotion features are then fed back into the corresponding convolutional pooling layers for updating, making the captured emotions more suitable for dialogue style recognition: It is a convolutional layer. It is a max pooling layer.

3. The video dialogue style recognition method based on multimodal emotion fusion according to claim 1, characterized in that: The multi-head attention network mentioned in step 9) is specifically as follows: Visual features with temporal information and visual emotional characteristics Fusion: 9.1) Input the visual features with temporal information into a linear layer to obtain the index Q, input the visual emotion features into a linear layer to obtain the key value K, and input the visual emotion features into another linear layer to obtain the value V: in Represents the weights of the linear layers in the index calculation. The linear layer representing the weighting of the computed index. The weights of the linear layer that calculate the key values. The emphasis on linear layers that represent the computation of key values. The weights of the linear layer representing the calculated values, The bias towards linear layers representing calculated values; 9.2) Multiply Q and K, then divide by a scaling factor, and input the result into a softmax layer to obtain a feature similarity matrix. Multiply the similarity matrix by V to incorporate emotional attention into the visual features. in It refers to the number of attention heads in a multi-head attention network. It is a scaling factor. It is a matrix transpose operation; 9.3) Obtain the multi-headed emotion attention characteristics The concatenated features are then fed into a linear layer for updating, yielding the complete emotion and attention features: in This represents the weights of the linear layer used to update the emotion attention features. This represents the bias of the linear layer used to update emotional attention features. Representative feature splicing operation; 9.4) The emotional attention features and visual features are added together and standardized, then fed into the feedforward network for updating. The sums are then added together and standardized again to obtain the final visual features incorporating emotional attention. in Representative feature standardization, Represents a feedforward network. , , and These are the parameters of the feedforward network; Similarly, auditory features with emotional attention can be obtained. and text features with emotional attention .

4. The video dialogue style recognition method based on multimodal emotion fusion according to claim 1, characterized in that step 11) is a weighted fusion method for multimodal results: in , and The classification results are for visual, auditory, and textual modalities. , and These are the weights of the visual, auditory, and textual results, representing the influence of different modal features on the final classification result, and are added to the training layer in the form of a linear layer.

5. The video dialogue style recognition method based on multimodal emotion fusion according to claim 1, characterized in that in step 1), the video is input into a pre-trained OpenFace model to extract visual features from the video. Step 2) Use ffmpeg commands to convert the video into audio, and then input it into a pre-trained librosa model to extract audio features. In step 3), the video is input into the pre-trained deepspeech model to extract text, and then the text is input into the pre-trained BERTTokenizer to extract text tokens. .

Images